issues with PI SD Card .. Failing to boot... and a Heartbeat between PI and Arduino

The more technical aspects of Spirit, and carry-over discussion from Kickstarter updates.
marckarasek
Posts: 23
Joined: Sun Jan 14, 2018 1:59 pm

issues with PI SD Card .. Failing to boot... and a Heartbeat between PI and Arduino

Postby marckarasek » Sat Feb 24, 2018 11:46 pm

Has anyone else seen an issue with the PI SD Card failing to boot and having to reformat the card?

I am concerned that there is no heartbeat between PI and Arduino to handle the shutdown properly.

I am going to post this here and also email Kevin. We need a heartbeat between the P{i and Arduino. This can be a GPIO dedicated to this. The PI strobes/heartbeats the line every X time. (time TBD). The Arduino monitors this line when the power button is pushed and it has sent the signal to the PI to shutdown. Once the heartbeat goes away, then it is ok to turn power off. Without this we run the risk of having the PI still up and shutting down when power is removed.

Thoughts, Concerns, etc..??

Marc

marckarasek
Posts: 23
Joined: Sun Jan 14, 2018 1:59 pm

Re: issues with PI SD Card .. Failing to boot... and a Heartbeat between PI and Arduino

Postby marckarasek » Sun Feb 25, 2018 3:03 am

I just ran a test with the PI running Stretch and attached to a HDMI monitor.

The Arduino was running the Spirit_BaseSketch_PiControl_Rev01 sketch. Hit power the led flashes for a few secs and then power down. The PI was still up at a screen no power down on the PI side.

This is a huge problem and will cause corruption on the SD card for PI.

:twisted: :evil: :oops:

gpvillamil
Posts: 75
Joined: Fri Jan 26, 2018 5:03 pm

Re: issues with PI SD Card .. Failing to boot... and a Heartbeat between PI and Arduino

Postby gpvillamil » Sun Feb 25, 2018 3:50 am

If you install the Python software from Kevin, you set up two tasks as cron jobs that start on boot: a PGPixelServer and a Shutdown monitor. https://plumgeek.gitbooks.io/spirit-rover-code-quick-reference/content/raspberry-pi-setup/pi-software-install.html

The latter monitors a GPIO pin and starts a shutdown sequence when it is pulled down.

When you press and hold the power button, the Arduino sets the GPIO pin, and starts rapidly flashing the power LED. This signals a 5 second delay to allow the RPi to shut down (if the Shutdown monitor is running).

You can confirm the Shutdown monitor is working by connecting to the Pi with SSH and pressing the power button, you should see the text “Spirit Shutdown Now!” in the console right before it shuts down. (It works for me)

marckarasek
Posts: 23
Joined: Sun Jan 14, 2018 1:59 pm

Re: issues with PI SD Card .. Failing to boot... and a Heartbeat between PI and Arduino

Postby marckarasek » Sun Feb 25, 2018 4:23 am

Ok I will try this...

We should really have a heartbeat over this gpio so that the PI can signal to the Arduino when it has shutdown..
I will check out the python code and see what I can do to add to it or make a kernel driver for it in C.

5 Sec is still not enough time for the PI to properly shutdown..

Looked at the script and while it is monitoring the pin (why does he set it as an output then input??) The shutdown is NOT happening.
The message you see is only one of two that should be on a console. This message is from the sudo wall "Spirit Shutdown Now!" cmd being executed.
There should be following this a message from the system saying it is going down. Actually is should show the same text again. Since you do not see two messages, the PI is NOT shutdown properly and this WILL result in SD corruption. Having just a timeout on one side w/o some kind of heartbeat/handshake from the other is a bad design and is only asking for trouble. For example suppose I have the full desktop, plus some background tasks running on the PI. It may take longer than 5 sec to shutdown.

My point is not all PI will take less than 5 sec to power down. SD card corruption means you need to remove and reformat and potential resetup (install SW, etc..) on the SD card. Which is a pain in the neck. Mostly due to the SD card placement on Rover.

I really wish Kevin would just put all the spec/docs out there. I took a look at the setup doc again and all it says is the code monitors a "special" pin. Not very technical and very frustrating.

The community could then just run with it and create wikis, docs, code and really run with it. Even if he we had to sign some kind of legal doc for IP reasons it would still be helpful..


i will attempt to fix this but without any technical docs, which pins are available? Is there an open GPIO I can use as a heartbeat? This will be a difficult task.

:ugeek: :geek: :o

gpvillamil
Posts: 75
Joined: Fri Jan 26, 2018 5:03 pm

Re: issues with PI SD Card .. Failing to boot... and a Heartbeat between PI and Arduino

Postby gpvillamil » Sun Feb 25, 2018 4:57 pm

Shutdown is occurring correctly.

Code: Select all

sudo shutdown -h now "Spirit Shutdown Now!"
will not generate a wall message. You can just try it! If you don't specify a time interval, the "system going down" message is not generated, and neither is any supplementary message text. This is standard documented behavior for the shutdown command.

I get your point about not being able to predict that shutdown will always take less than 5 seconds. However, generating a heartbeat that persists until actual shutdown means that it can't run in userspace, and requires kernel changes. Seems risky to me.

It seems that management of the power button is handled by the PIC, not the Arduino. I wonder if a way of managing this is to take a reading of current level when power button is pressed, toggle the GPIO line to let the RPi know that shutdown is imminent, and then monitor current levels for a sudden drop in power consumption that says the RPi has actually shut down.

marckarasek wrote:There should be following this a message from the system saying it is going down. Actually is should show the same text again. Since you do not see two messages, the PI is NOT shutdown properly and this WILL result in SD corruption. Having just a timeout on one side w/o some kind of heartbeat/handshake from the other is a bad design and is only asking for trouble. For example suppose I have the full desktop, plus some background tasks running on the PI. It may take longer than 5 sec to shutdown.

Kevin
Posts: 181
Joined: Tue Jul 28, 2015 12:56 am

Re: issues with PI SD Card .. Failing to boot... and a Heartbeat between PI and Arduino

Postby Kevin » Sun Feb 25, 2018 8:30 pm

gpvillamil wrote:Shutdown is occurring correctly.
It seems that management of the power button is handled by the PIC, not the Arduino. I wonder if a way of managing this is to take a reading of current level when power button is pressed, toggle the GPIO line to let the RPi know that shutdown is imminent, and then monitor current levels for a sudden drop in power consumption that says the RPi has actually shut down.


Good question and excellent follow-up idea.

The power system is controlled by the PIC. It holds a line that keeps the main battery switch turned on. When this goes away, the power is shut down to the entire robot.

The delay used by the PIC can be adjusted from the Arduino with the PIC_SetShutdownDelay function. (Look it up in the Comms tab of the Arudino sketch). This would need to be called in the startup function of Arduino as this register in the PIC defaults back to the normal value after each power off (there's no eeprom or anything in the PIC). This would allow you to set it to a delay that is plenty long.

Now I really like the idea of monitoring current and watching for a drop. However the more I think about it, I don't think it's possible to carry this out the way the firmware is in the PIC. It's been a long time since I looked but I don't think there is any command that can shut off the PIC, so it would be hard to monitor this. I wish I'd have thought about that when writing that code.

We considered a few ways to do this and didn't want to force users to watch a heartbeat or anything as that over complicates things for new users (like if the Arduino had to constantly watch and respond to a signal or something).

I'm going to post the main page of the schematic that shows all pins and processors in the next few minutes.

marckarasek
Posts: 23
Joined: Sun Jan 14, 2018 1:59 pm

Re: issues with PI SD Card .. Failing to boot... and a Heartbeat between PI and Arduino

Postby marckarasek » Sun Feb 25, 2018 9:16 pm

The heartbeat is just a kernel module, this could be a standalone module or incorporated into another one. It is no big deal. Have done it multiple times in the past.

If you do not want to always have to run your code as sudo or some other such nonsense you have to start lookign at kernel modules. Besides this is the "correct" way to do hardware anyway, doing calls via ioctl or standard driver apis.

You can get a look at an example of this in my project (https://github.com/MarcKarasek/rpi_rgb_ ... rneldriver) that has a driver for talking to a PI hat to control a 32x32 led display. It uses iotcl calls to send in data. I ported the driver over from a userspace program that had to be run as sudo each time.

I will put together a simple kernel module for this and post the code to the repos I have setup.
See other post : viewtopic.php?f=18&t=8054
Hopefully I can find a spare gpio or use the one already in use gpio17 (i think) ..

Kevin
Posts: 181
Joined: Tue Jul 28, 2015 12:56 am

Re: issues with PI SD Card .. Failing to boot... and a Heartbeat between PI and Arduino

Postby Kevin » Sun Feb 25, 2018 9:29 pm

Okay guys and girls. Just posted the schematic of all the connections between the 3 processing centers.

I did think of a way to get the PIC to shut down. It would be possible to improve the shutdown sequence based on seeing current drop off (when the Pi shuts down). Someone would have to give it some thought, but I think it would be possible.

This is going to be a fairly complex process though, so it may be better to just adjust the delay the PIC waits before pulling the power. If the default of 5 seconds isn't long enough, you can use PIC_SetShutdownDelay function in the setup function of your Arduino to set this shutdown delay to a longer amount.

If you want to attempt something more complex, this will be helpful.....

The PIC monitors battery voltage, and if it drops too low, it shuts down. This is to prevent completely discharging the battery if the unit is left on. You can adjust this threshold with the Arduino function PIC_SetVoltageEmgShutdown, which targets PIC register 107. Normally this level is set to 3.2 volts (3200 millivolts).

You could use this function to set the emergency shutdown to something like 4.2 volts and assuming batt voltage is below that at the time, it should cause the PIC to shut down the system. I don't think there is anything to trap the value to a certain level, so I'm pretty sure you could set this shutdown value to like 5 volts then you should guarantee that you trigger the shutdown as a fully charged battery is about 4.2 volts.

You can do the same thing with the emergency shutdown current. For this use the Arduino function PIC_SetCurrentEmgShutdown, which targets PIC register 105. A shutdown due to over-current will happen if the current draw is above this level. So you could set the emergency shutdown current to something really low like 10 milliamps, which should trigger the shutdown.

===========

To make this all work, I think you'd want to catch the button press and somehow communicate to the Pi that it should begin shutdown via a process on the Arduino. Then the Arduino can sit and measure current until it drops off, then the Arduino can flip that voltage or current threshold to the new out-of-bounds value which should cause the PIC to kill the main power switch.

You can query the position of the PWR and BTN buttons via the Arduino with PIC_ReadInputs, which will update the global variables PIC_Button and PIC_Button_Pwr. Just keep in mind this causes I2C traffic which normally is okay, but if you have the Pi blasting the I2C bus with lots of traffic this could cause the bus to hang.

Hopefully that's helpful. :)

marckarasek
Posts: 23
Joined: Sun Jan 14, 2018 1:59 pm

Re: issues with PI SD Card .. Failing to boot... and a Heartbeat between PI and Arduino

Postby marckarasek » Mon Mar 12, 2018 1:20 pm

I think I have an alternative solution.

We can do a heartbeat over the I2C interface used to send cmds from PI -> Arduino. It would just be a command that is sent every 30 sec or 1 min. It refreshes a global on the Arduino side. When the Arduino sets the power down, it waits for X secs and if the global is not reset it powers down the rover.


Who is online

Users browsing this forum: No registered users and 1 guest