CCS C Software and Maintenance Offers
FAQFAQ   FAQForum Help   FAQOfficial CCS Support   SearchSearch  RegisterRegister 

ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

CCS does not monitor this forum on a regular basis.

Please do not post bug reports on this forum. Send them to CCS Technical Support

Saving data to SD memory card
Goto page 1, 2  Next
 
Post new topic   Reply to topic    CCS Forum Index -> General CCS C Discussion
View previous topic :: View next topic  
Author Message
jlucas



Joined: 08 May 2011
Posts: 41
Location: Carthage, MO

View user's profile Send private message Visit poster's website

Saving data to SD memory card
PostPosted: Wed Jan 04, 2012 2:49 pm     Reply with quote

I’m using a PIC 18F4520 in a GPS vehicle tracking application. I’m collecting data via radio from moving vehicles.

I want to use a micro SD card for event logging. I know there’s a TON of information on this site about how to do this. That’s my problem – information overload. Perhaps if I can get answers to a few questions, I can get unstuck.

The first regards the hardware interface. I don’t need to read anything back to the PIC from the SD card. Can I get by with 3 wires to the SD card: data in, +3.3 V, and ground? This would save having to add a 3.3 to 5 V level translator to up-convert the voltage for data back to the PIC.

Moving on to the firmware, first I would like to create a folder named with the MAC address of the RF module. Then I would like to save the data from each event to a separate file named with the time and date of the event. I can use csv, tab-delimited, or fixed-field-length format, as long as I can pull the SD card, stick it into a PC, and open the file in Excel.

I have enough memory in the PIC to buffer the data for an entire event, then create and write the file to the SD card after the event is over. Can someone point me to some information on how to 1) create a folder, 2) create a file, and 3) write data to that file? The data will contain both text and numbers as well as the field separator and a Return at the end of each 1-sec record.

Some other questions:

Can I use SD cards preformatted with FAT32?

Is there any limit to the size of the SD card? I'm thinking of using 2-8 Gb.

Can I create and name a folder on the SD card to hold all the files?

Can I use long file names?

Can the SD cards be hot-pluggable/unplugable?

I don’t expect you guys to write my code for me, but if anyone can point me to a source of help for this, I would be most appreciative.

Thanks much,

Jim
_________________
Always remember, things are never so bad that they can't get worse.
temtronic



Joined: 01 Jul 2010
Posts: 9255
Location: Greensville,Ontario

View user's profile Send private message

PostPosted: Wed Jan 04, 2012 3:33 pm     Reply with quote

Short answer.....use another chip for data logging.

Longer answer.. You'll spend a LOT of time researching/looking for code, cutting code, R&Ding a workable solution using mismatched devices(5V vs 3V). Yes, most , if not all of the 'code' has been cut but you'll have to work out the nasty details and time is money, don't kid yourself.

Longest answer...
There is an EASY alternative to SD cards for data logging. Use a FTDI Vinculum module. For $30 out the door you get a 5v PIC compatible USB flash drive data logging module.EASY serial command/data transfers.You can be 'up and running' in less than a 1/2 a day. Been there doing that.

Once you figure out the record format for storing data, it's as easy as one print command.Data gets stored as CSV format making it a piece of cake to import into a spreadsheet.Currently I have one recording data at 115200 baud, zero problems,about 200 bytes of data per record,every second,24/7.

Ok, so it's a 'big' chunk of money up front for the module but you get instant compatabilty with 5V PICs, simpler serial data transfers,huge storage,and all the messy FAT,files,dir,etc is done for you so you don't need to have 'SD drivers' in the PIC, saving space,more room for 'features' .Consider what your R&D cost is in 'engineering labour'. For less than 1 hour( @$50 /hr) you can have a viable, working solution up and running.Work smarter..not harder.
temtronic



Joined: 01 Jul 2010
Posts: 9255
Location: Greensville,Ontario

View user's profile Send private message

PostPosted: Wed Jan 04, 2012 3:33 pm     Reply with quote

Short answer.....use another chip for data logging.

Longer answer.. You'll spend a LOT of time researching/looking for code, cutting code, R&Ding a workable solution using mismatched devices(5V vs 3V). Yes, most , if not all of the 'code' has been cut but you'll have to work out the nasty details and time is money, don't kid yourself.

Longest answer...
There is an EASY alternative to SD cards for data logging. Use a FTDI Vinculum module. For $30 out the door you get a 5v PIC compatible USB flash drive data logging module.EASY serial command/data transfers. You can be 'up and running' in less than a 1/2 a day. Been there doing that.

Once you figure out the record format for storing data, it's as easy as one print command.Data gets stored as CSV format making it a piece of cake to import into a spreadsheet. Currently I have one recording data at 115200 baud, zero problems, about 200 bytes of data per record, every second, 24/7.

Ok, so it's a 'big' chunk of money up front for the module but you get instant compatabilty with 5V PICs, simpler serial data transfers,huge storage,and all the messy FAT, files, dir, etc is done for you so you don't need to have 'SD drivers' in the PIC, saving space,more room for 'features'. Consider what your R&D cost is in 'engineering labour'. For less than 1 hour( @$50 /hr) you can have a viable, working solution up and running. Work smarter..not harder.
asmboy



Joined: 20 Nov 2007
Posts: 2128
Location: albany ny

View user's profile Send private message AIM Address

PostPosted: Wed Jan 04, 2012 4:18 pm     Reply with quote

i use this FABULOUS module as a backup data logger in several different instruments i've designed.

works like a charm with up to 4GB micro SD cards i'ved tested - and is very fault tolerant - as it does a FREQUENT buffer commit to any open file you are writing too-power failure has never to my knowledge corrupted a disk either

http://www.sparkfun.com/products/9530

attaches to the TX -// or RX pins of a pic EZ 2
we buy them by the 100's and at $20 bux - is a bargain for what it can do
ckielstra



Joined: 18 Mar 2004
Posts: 3680
Location: The Netherlands

View user's profile Send private message

PostPosted: Wed Jan 04, 2012 4:20 pm     Reply with quote

Why is it that so many people in this forum want to interface a 5V PIC to a 3.3V SD card?
For making this work you will need a 3.3V to 5V translation because data can only be written to the card when you check the response answer from the card.
Some work arounds for this problem are possible but none of them brilliant.
My advise, use another PIC that can run at 3.3V directly and you won't have to invest in the extra cost for voltage converters and second power supply. As a bonus you save some power as well. For example the 'LF' version of your current PIC of one of the newer models.

Then about you saying you have enough memory available; you realise that writing data to the SD card happens in blocks of 512 bytes so the FAT library might use more of your 1536 bytes RAM than you anticipated.

About available FAT libraries:
- CCS supplies one for free. It works but some problems have been reported with the format function (it doesn't write a correct partition table or something like that causing Windows not being able to read the files). Once the CCS format is used the effect can only be undone by hard writing data to card address 0-512.
- The code library of this forum has at least two free implementations. Check the threads for comments on quality.
- Andrew, nickname asmallri, is a member of this forum and sells a commercial library with good reviews.

Quote:
Can I use SD cards preformatted with FAT32?
Yes
Quote:
Is there any limit to the size of the SD card? I'm thinking of using 2-8 Gb.
SD-cards go to a maximum of 2Gb (some 4Gb).
Larger capacity cards are of the SDHC (High Capacity) type with a slightly different protocol. None of the above mentioned free libraries supports SDHC cards, Andrew's library does.


Quote:
Can I create and name a folder on the SD card to hold all the files?
I should think so, up to the limits imposed by FAT32: Maximum 65,536 entries with 8.3 names. Longer file names will 'eat up' additional entries and so reducing the maximum file count.

Quote:
Can I use long file names?
With most libraries, yes.

Quote:
Can the SD cards be hot-pluggable/unplugable?
No, during the short data write actions the card should be left in place.
What you could do is to create all files beforehand with maximum size. Then pulling the card during a write action would only corrupt the file being written but leave the directory structure and other files intact.


For a one time off commercial product Temtronic's suggestion will save you a lot of time and money, it might even be cheaper than buying a good FAT32 library.
jlucas



Joined: 08 May 2011
Posts: 41
Location: Carthage, MO

View user's profile Send private message Visit poster's website

PostPosted: Wed Jan 04, 2012 6:28 pm     Reply with quote

Thanks, everybody!

temtronic, at $30 per unit, the FTDI Vinculum module is rather expensive compared to the SD card and holder. Since I’m hoping to sell these thing by the 1000s, it's worth spending $1000 on R&D if it saves me $1 per unit. It still might be the best solution, though.

asmboy, your $20 solution looks intriguing. It’s still more than I would like to spend, but it’s definitely on the table at this time.

ckielstra, I guess because they don’t make SD cards that run 5V like everything else on my board. I can get down to 3.3 V with resistors. Up-converting requires another part, but that should be less than $1. If I can get by without additional hardware other than that, the card, and card holder, I'll be money ahead.

I’ll only need one or at the most two 512-byte blocks of data for each file, and once it’s written to the SD card, I don’t need to keep it in RAM. If the data goes over 512 bytes, I guess I can write it in two operations and append the second block to the file.

A 2-Gb SD card should be plenty.

File names longer than 8.3 would be very nice; I can get by with much fewer than 65,536 entries.

By hot plugging and unplugging, I mean while the unit is under power, but not during a write operation.

Again, many, many thanks to all. I’ll get to work studying these new possibilities.
_________________
Always remember, things are never so bad that they can't get worse.
temtronic



Joined: 01 Jul 2010
Posts: 9255
Location: Greensville,Ontario

View user's profile Send private message

PostPosted: Wed Jan 04, 2012 7:20 pm     Reply with quote

The Vinculum module I was talking about is a complete unit.VNC1L chip, PCB, USB connector,misc resistors,caps, headers,etc. designed for rapid prototyping/ proof of concept. Chip alone can be bought for about $7.50 /1k.

Your math is interesting. If you only save $1 /unit and sold 1000 units that's $1,000 you saved but it cost you $1,000 in R&D so you broke even,moneywise. Time is a critical factor in getting a product to market.That $1,000 R&D money is only 10 hrs for me(call it 2 days) and frankly there is no way anyone can cut code equal to what the Vinculum can do in that time frame.
We all 'know' that a bigger PIC is better but most try to squeeze 'everything' into the cheapest unit trying to save a few pennies meanwhile R&D time has consumed 100s if not 1,000s of $.Same thing applies to say voltage regulators, while a 78L05 may do the job(on paper) using a 7805 can easily save on repair costs,service calls,etc. as it'll take 'abuse' better.

Just something to consider....
jlucas



Joined: 08 May 2011
Posts: 41
Location: Carthage, MO

View user's profile Send private message Visit poster's website

PostPosted: Wed Jan 04, 2012 10:30 pm     Reply with quote

Thanks again, tt. I have some work ahead of me now. Wink
_________________
Always remember, things are never so bad that they can't get worse.
ckielstra



Joined: 18 Mar 2004
Posts: 3680
Location: The Netherlands

View user's profile Send private message

PostPosted: Thu Jan 05, 2012 7:20 am     Reply with quote

Why spend money to get the SD card to interface with 5V and not run everything else at 5V? I'm wondering which 5V components you have in your design that are not available in a 3.3V version.

For example you could replace the 18F4520 chip by a 18F45K20 which has a lot more features, can run up to 64MHz at 3V (versus 25MHz at 3.3V and 40MHz at 5V for the older 4520). Is also cheaper because produced with newer teechnology ($1.60 v.s. $3.29 at high volumes).
temtronic



Joined: 01 Jul 2010
Posts: 9255
Location: Greensville,Ontario

View user's profile Send private message

PostPosted: Thu Jan 05, 2012 8:15 am     Reply with quote

did a check at digikey and the Vinculum-II version is about $3!
asmallri



Joined: 12 Aug 2004
Posts: 1636
Location: Perth, Australia

View user's profile Send private message Send e-mail Visit poster's website

PostPosted: Thu Jan 05, 2012 9:37 am     Reply with quote

jlucas wrote:
Thanks, everybody!

temtronic, at $30 per unit, the FTDI Vinculum module is rather expensive compared to the SD card and holder. Since I’m hoping to sell these thing by the 1000s, it's worth spending $1000 on R&D if it saves me $1 per unit. It still might be the best solution, though.

asmboy, your $20 solution looks intriguing. It’s still more than I would like to spend, but it’s definitely on the table at this time.

ckielstra, I guess because they don’t make SD cards that run 5V like everything else on my board. I can get down to 3.3 V with resistors. Up-converting requires another part, but that should be less than $1. If I can get by without additional hardware other than that, the card, and card holder, I'll be money ahead.

I’ll only need one or at the most two 512-byte blocks of data for each file, and once it’s written to the SD card, I don’t need to keep it in RAM. If the data goes over 512 bytes, I guess I can write it in two operations and append the second block to the file.

A 2-Gb SD card should be plenty.

File names longer than 8.3 would be very nice; I can get by with much fewer than 65,536 entries.

By hot plugging and unplugging, I mean while the unit is under power, but not during a write operation.

Again, many, many thanks to all. I’ll get to work studying these new possibilities.


An additional data point, Microsoft holds a patent on the FAT file system pertaining to the integration of long file names in the FAT data structure. To avoid infringing on this patent, the Brush Electronics' file system driver (my driver) does not support long file names.

If you are using a FAT file system you MUST have more than 512 bytes allocated to the file system. You may think that performing a write in multiple parts would avoid this issue but it is not the case. This is because when you cross a cluster boundary the directory entry must be updated, this required reading the sector containing the entry (512 bytes) into a buffer, modifying the appropriate entry, then writing out the entire 512 byte buffer.

If you must use a PIC18F processor then ideally select one with plenty of RAM such as the PIC18F46K20 (or derivative). If you are producing 1000s of units then please check out my file system driver, it comes with a fully functional data logger application demonstrating the decoupling of data acquisition from logging and provides a DOS like command line interface. The driver with the supplied sample applications will decrease your software and hardware development time, and will result in lower per unit cost as it does not require additional hardware other than the SD/MMC card socket.
_________________
Regards, Andrew

http://www.brushelectronics.com/software
Home of Ethernet, SD card and Encrypted Serial Bootloaders for PICs!!
jlucas



Joined: 08 May 2011
Posts: 41
Location: Carthage, MO

View user's profile Send private message Visit poster's website

PostPosted: Thu Jan 05, 2012 12:59 pm     Reply with quote

Quote:
Why spend money to get the SD card to interface with 5V and not run everything else at 5V?

I've already built 100 boards. I have a cable header on the board with +5V, ground, and several I/O connections, so I can add a small PCB to hold some additional components, but I can't change the MCU at this time.

Changing to a 3.3V chip in the next turn would involve a major redesign of the board. I might do that when the time comes, but I have to get the first 100 units out the door first.

Quote:
did a check at digikey and the Vinculum-II version is about $3!

Now we're talking!

asmallri, I will try to keep all that in mind.

Whew! Who knew something that on the surface appears to be so simple could be co complicated? Confused
_________________
Always remember, things are never so bad that they can't get worse.
asmboy



Joined: 20 Nov 2007
Posts: 2128
Location: albany ny

View user's profile Send private message AIM Address

PostPosted: Thu Jan 05, 2012 1:19 pm     Reply with quote

Quote:

asmboy, your $20 solution looks intriguing. It’s still more than I would like to spend, but it’s definitely on the table at this time.


Because my design consultancy is small ( ME, ++ a layout genius , and 3 part time techs && a gal friday) -the issue of BUILD || buy is always present. a big consideration is to use hardware or modules - wherever reasonable - to avoid delays in the time to market issue.
( ok - lets say say l time to client payoff )

i don't know about you - but having to NOT debug or worry about interactions in my PIC code is worth a lot to me.

things like Openlog , or the $3 simple FTDI UM232 rs-232/USB 2 bridge chip - these things make my PIC work so much simpler and robust - compared to using a PIC with its own USB COMM or supporting the NASTY overhead of dealing with a naked MMC module - that i rarely think of how to load MORE code into the pic -
but how to write LESS -
and let the hardware do more ;-))
jlucas



Joined: 08 May 2011
Posts: 41
Location: Carthage, MO

View user's profile Send private message Visit poster's website

PostPosted: Thu Jan 05, 2012 1:33 pm     Reply with quote

Sounds like good advice, asmboy. I have a lot of good information to work with now. Thanks!
_________________
Always remember, things are never so bad that they can't get worse.
n-squared



Joined: 03 Oct 2006
Posts: 99

View user's profile Send private message Send e-mail Visit poster's website

PostPosted: Fri Jan 06, 2012 5:42 am     Reply with quote

Quote:

Up-converting requires another part, but that should be less than $1.

You do not need up-converting conditioning since the PIC will consider 3.3V signal to be a "1".

BR
NN
_________________
Every solution has a problem.
Display posts from previous:   
Post new topic   Reply to topic    CCS Forum Index -> General CCS C Discussion All times are GMT - 6 Hours
Goto page 1, 2  Next
Page 1 of 2

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © 2001, 2005 phpBB Group