Parsing a lot of data and then plotting it, dopamine in rat brains

Hi I am doing research on dopamine levels/pulses in a rats brain. Carbon fiber electrodes are put into the brain and that acts as a sort of battery. There is a baseline voltage curve that is somewhat constant in the rats brain. When dopamine is release, the voltage spikes. By subtracting the baseline voltage from the data collected, you are able to see when dopamine is released. I have 16 channels going to the brain. Every tenth of a second, data is collected from the channels going 1 through 16 1 through 16 1 through 16 and so on. There is a high resolution clock that helps to time stamp the data collection. The environment in the rats brain is always changing, so a new baseline needs to be configured every 50 seconds or so. I have written code so it parses the 16 channels into separate matrices, but I need to subtract the data from a new baseline that is determined every 50 seconds.

After the data is collected I need to plot it, comparing it to a horizontal line of 0.6V. If I plot it all from left to right, then the graph would go on for miles, so I need the graph to plot from left to right for ~50 seconds then skip down below the plotted line and plot another right below it just the same for ~50 secs then plot another right below it for ~50 secs and keep doing this until it reaches 24hrs.

After 24 hrs of collecting data the data is about 20GB so its a hearty chunk of data. Please let me know if you have any insight on how to go about solving/writing this code. Especially the graphing part of it. Would also like to know if there is a toolbox that I could use to make this all easier. Thank you for your help

[dpb]

At 1 minute, say, instead of 50 sec, there are 24*60=1440 traces in a 24-hr day. Having all of those on one figure is going to just not be possible; there aren't enough pixels on a monitor of any size to show them all.

You can probably do from a couple to several hours at a time before just run out of real estate; will depend on just how dynamic the traces are and what level of resolution you'll need to be of any value at all.

For visualization, might want to look at waterfall to put a number on a given axes with some spacing between them rather than just straight traces--experimentation with some of the data will be the only way to really know what "works" and what doesn't...I presume there have been other papers published in the subject area, looking at some of their data presentation methods may help provide ideas as well.

As a note, I'd think this baseline shouldn't be just blindly recomputed every N seconds independently, but should at least be smooth function with time itself as well. Smoothing splines could perhaps be useful here.

It'll be easy enough to bring in sections of the data at a time, the challenge will be in just how much you can actually realistically do something with simultaneously although it'll be relatively simple I'd think to do the calculations, being able to display them will undoubtedly be a challenge.

Of course, when you have this much data, just how much you can physically look at is limited as well; you'll have to develop procedures to automate retrieving whatever measures there are in it that are of interest.

[Patrick Lydon]

You are correct on most of these points. I don't think I will ever be showing 24hrs of data on one graph, I will be just taking certain time frames of it, I am sorry my wording was poor. I have been using smoothing splines to display my data over the past couple years. I have not seen someone graph it the way I want it to be graphed. Thank you for your response. I will look more into the waterfall plotting system. If you have any more points for me to consider, please let me know. Or any other advice about the Waterfall plot.

[dpb]

If you have examples of what you think you would like the plotting to look like, attaching them even as hand drawn sketches could help folks visualize what you're thinking -- otherwise we can only guess what that is which may not be even close! :)

Attaching a .mat file with some of the data would let folks here play some and get an idea at least of what its general characteristics are. Often someone will see something that strikes their fancy and do quite a bit of interesting work...but can't really do anything more than such generalities above w/o data.

Of course, if you haven't already, study the documentation sections on animation and performance in graphics; most of those techniques will be useful in what you're after.

Waterfall plots are used a lot in vibration analysis and the like with power spectra to show the evolution of a given feature set with time; I'd think that could be helpful in such data as you have altho it may be that there isn't such a spatial relationship given it's an animate subject rather than mechanical or the like. It is a way to cram a lot of data on a given axis providing there is some semblance of correspondence between traces so doesn't just turn into a blob. What will happen w/ your data I've no klew, just probably worth taking a look at the data to see...

[Patrick Lydon]

Thank you for all of the help dpb. I will upload 10s worth of data so others can get a better idea of where I am coming from. For a 10s piece of data: a waveform (which is ~500 points) is created by each channel (there are 16 channels) every 0.01s. So for ~10s, each channel has a matrix of data points (100,500), 100 being the time and 500 being the amount of points in one waveform. So essentially it is a matrix of 100 waveforms that I must plot. So in the end I have 16 plots that I must make (one for each channel). So, when I was looking up the waterfall plot, I see that the form is waterfall(X,Y,Z). At first I thought this might not work since, I dont necessarily have a Z. But then I read more about it and was thinking that my Z could be 0.6V? I am somewhat confused about this plot. If you could offer some clarification that would be fantastic.

[Patrick Lydon]

So I tried plotting my data in a waterfall plot (I will attach a picture of the plot). There are a lot of vertical lines, I would like it to be smooth and make a nice wave-like look. How do I do this?

[dpb]

I'm having hard time wrapping head around this, sorry... :)

There seems to be two disparate times here...I'm having trouble understanding where the 2D arrays come from for each channel; my simple-minding thinking says there ought to a a waveform of T/dT samples for each of 16 channels from some initial T0. What am I missing?

I'm trying to figure out who to plot against whom to be comparing the evolution of the same waveform with time or the various channels over a given time. It seems there's an extra dimension here I'm not following.