Remote debugging

[guy]

Does anyone know of some kind of remote debugging / cellular / wireless debugging tool ?
Rarely I get complex projects (this time it's RF mesh networking) which act differently in the lab and in the field. Having such a tool and shipping it with the system until it is fully debugged could be great.

I am also considering writing my own 'hack': On one side there will be a CCS C library that will be compiled together with the source file into the target chip. It will include a bootloader and some basic debugging features. It will communicate through a UART (without interrupts, very basic) and can be connected to any serial comm. device - modem, wireless, serial to USB etc.
On the other side of the link is a simple app for the debug interface, and recompiling/programming will be done using whatever IDE the user chooses.
The bare minimum is bootloader + monitoring data that will be dumped to the serial port + Remote reset.
Breakpoints and a debug executive can be accomplished by placing the whole debug executive code inside an interrupt that will be manually called by the user at the breakpoint (set the int.flag of an unused interrupt). It can read/write to RAM, compare values, do a pass-count, or resume execution.

[temtronic]

You've got 3 potential problems to deal with
1) hardware
2) software
3) firmware

1) somehow your 'remote' operation has to overcome a hardware 'issue'. This means it'll have to test 'low level' functionality of the PIC and the rest of the product,so you need a lot of 'feedback' to confirm the hardware is 100%. EVERY I/O pin needs to have feedback to confirm it's OK....

2) the actual running of the program will need some 'checks and balances', verification that the program is operating properly(ie : no bad ADC results...)

3)the actual 'remote' program has to have rock solid communications with feedback. If you send a command code it HAS to be 'bullet proof'. This means using checksums for all data transfers and again feedback that 'stuff was done' properly

The major stumbling block is time and codespace. Just cutting basic 'POST' code could take weeks or months depending on PIC, peripherals and the actual program. There's also the space needed to include this 'test code' within the product.Perhaps an external EEPROM is required as the time to transfer a 'test' program could be horendous.As well, the product will need 'feedback' to confirm that I/O ports were set as required,which means additonal hardware,so more chips, more PCB area, more time, more cost.... It can get very,very complicated FAST.

Most of my 'worked in lab..failed in the Real World' problems have been due to EMI 'attacks',solved by 'hardening' of EVERY connection to the Real World. LOTS of caps, chokes,shielding, beefier power supplies,better PCB design like heavier traces, routing and parts layouts. I designed my own, self-indicating SSRs for my energy control systems decades ago. Just because the code turns on an SSR doesn't mean it actually did,you NEED to confirm it IS on.

It's best to tackle the hardware first,usually solves 99% of the problems. PICs themselves are reliable,so providing the code was solid in the lab, it's generally solid in the Real World.


just some point to ponder
Jay

[guy]

You bring up important points Jay. Thank you. I totally agree.
I assume both of us have many years of experience and by looking at 'odd' behavior we already have hunches where the problem lies, and we probably know the weak parts of the system even before it is built... This is the base assumption of this debugger - it will not be half as good as having the product in the lab with all measurement and test tools around, but it's Better Than Nothing. In fact I would call it BTN
Your ideas point directly to an external powerful PIC that will handle EVERYTHING except what has to be done inside the target PIC:
Debugger PIC: Memory (RAM+flash/external), comm. with modem, http stack (built into modem), self-sustainable, verification of comm., retry mechanisms WDT etc., relays & inputs for flexible I/O control, monitoring, and reset to the target. [ edit ] Oh, and RTC with backup battery to time-stamp the comm. with the target.
Target PIC: bootloader, send out register contents, signal breakpoint to debugger, receive commands (change register, reload code). My wild guess is 2-4KB of efficient C code in the target, assuming the debugger takes upon itself as much of the work as it can.

Re network stacks nowadays most cellular modems have built in stacks for HTTP get/put, FTP, etc. I already implemented those with several types of modems.