End result of my current project...

Well... It seems that I need to say goodbye to my current thesis research. I worked on a set of database integration algorithms. I'm not going to get into the nitty gritty derails here but in the end, I found that the competing algorithm gave me similar results, but was much faster.

I was just running some tests, scanning two tables with around 1000 rows each to generate similarity scores between columns. Our algorithm was running for around 4 hours. The competing algo finished in 5 minutes. Results look similar. I will be doing some data analysis on the result sets tomorrow but it does not look good... It's not a surprise really. I knew we were slow, but this big dataset just shown me how badly does our algorithm scale compared to the other one.

I will miss this project, but it seems that I would have to do some really heavy optimization to even get close to the other algo performance. Either that or figure out a convincing reason why our data is more accurate than the other one. In other words this might have been a blind alley all along - and I might have spent the next one or two semesters futzing with the data to make it look like we accomplished something.

Perhaps it will not be so bat to start fresh with a new project... Sigh... It still pisses me off that I wasted most of this summer. But oh well... There's not much that I can do about this right now.

Tags: thesis, masters, research, life sux