The major thing I wonted to cover in this post was how to get the separate files we would be receiving from our affiliates into data tables. The types of files we will received for this project are both CSV (Comma Separated Values) as well as XML (Extensible Markup Language). I wanted the end result to be the same type of object so that the same class could handle the processing algorithms without having to use overloaded methods or any type of polymorphism. Getting them to data tables seemed like a fairly viable solution because then I would only need to know which columns to grab based off which affiliate we were currently processing; information which is stored in the database with all of the affiliate information.

Getting these files to a data table actually ended up being a fairly easy task once I sat down and thought about it as well as look through some online articles about common ways of handling this type of information. Having worked with XML a lot in the past, I was confident this should not be hard. Let’s start taking a look at how it works. Lets start with the CSV file.

public DataTable ParseCSV(string path)
{
    if (!File.Exists(path))
        return null;
    string full = Path.GetFullPath(path);
    string file = Path.GetFileName(full);
    string dir = Path.GetDirectoryName(full);
    string connString = "Provider=Microsoft.Jet.OLEDB.4.0;"
    + "Data Source=\"" + dir + "\\\";"
    + "Extended Properties=\"text;HDR=No;FMT=Delimited\"";
    string query = "SELECT * FROM " + file;
    DataTable dTable = new DataTable();
    OleDbDataAdapter dAdapter = new OleDbDataAdapter(query, connString);
    try
    {
        dAdapter.Fill(dTable);
    }
    catch (Exception ex)
    {
        WriteError(ex);
    }
    dAdapter.Dispose();
    return dTable;
}

As you can see, not a whole lot to it. It simply gets passed a file location on the hard drive, gets the file information, and builds a OLEDB request setting the data source as the directory. Then we select * from the file which we can throw the results into a data table. Then we just run it through the rest of the class for it to process each row to see if the store exists, the coupon exists, and that the coupon is active and create database record where necessary.

StringReader XMLReader= new StringReader(filePath);
DataSet XMLTables= new DataSet();
theDataSet.ReadXml(XMLReader.ReadToEnd());
DataTable dtValues = XMLTables.Tables[Convert.ToInt32(intTableDepth)];
return dtValues;

This was the one I didn’t realize would be as easy as it was. When the file is passed through the ReadToEnd it parses the information into a data set, each child set having it’s own table. All that is needed to parse the XML properly is to know how deep the data is housed within the XML (the child nodes that house each item). In this case, I have a function that figures it out and passes it to this function when it’s called.

Now, all that needs done is to pass the data table to the functions that iterate through the information and collect what is needed. That pretty much sums up that part of this project. Thanks for reading.

        private void cmdParse_Click(object sender, EventArgs e)
        {
            if (txtInput.Text != String.Empty)
            {
                tvXMLView.ResetText();
                FillTree(tvXMLView);
            }
        }

This is just called when the user clicks a button and makes sure the text box is not empty. We are going to omit the code about adding the tab and tree view to the form, so the next step is to run the FillTree function.

        private void FillTree(TreeView currenttree)
        {
            try
            {
                XmlDocument dom = new XmlDocument();
                dom.LoadXml(txtInput.Text);
                currenttree.Nodes.Clear();
                currenttree.Nodes.Add(new TreeNode(dom.DocumentElement.Name));
                TreeNode tNode = new TreeNode();
                tNode = currenttree.Nodes[0];
                AddNode(dom.DocumentElement, tNode);
            }
            catch (XmlException xmlex)
            {
                MessageBox.Show("XML Malformed. Reason for Error: - " + xmlex.Message, "Error");
            }
            catch (Exception ex)
            {
                MessageBox.Show("Non XML Error Occured: " + ex.Message, "Error");
            }
        }
 

First we create an XmlDocument that we can use to read through the XML in the text box. Next we ensure the tree view is empty and we are not going to append more data to an already parsed tree view. Then we begin to iterate through the values and processing them through the AddNode function.

        private void AddNode(XmlNode inXmlNode, TreeNode inTreeNode)
        {
            string strCheck;
            XmlNode xNode;
            TreeNode tNode;
            XmlNodeList nodeList;
            int intLength;
            int intStart;
            int intEnd;
            int i = 0;
            if (inXmlNode.HasChildNodes)
            {
                nodeList = inXmlNode.ChildNodes;
                for (i = 0; i <= nodeList.Count - 1; i++)
                {
                    xNode = inXmlNode.ChildNodes[i];
                    inTreeNode.Nodes.Add(new TreeNode(xNode.Name));
                    tNode = inTreeNode.Nodes[(int)i];
                    AddNode(xNode, tNode);
                }
            }
            else
            {
                if ((inXmlNode.OuterXml).Trim().Contains("></"))
                {
                    strCheck = (inXmlNode.OuterXml).Trim();
                    intStart = 1;
                    intEnd = strCheck.IndexOf("></") + 3;
                    intLength = strCheck.Length - intEnd;
                    strCheck = strCheck.Substring(intStart, intLength - 1);
                    inTreeNode.Text = strCheck;
                }
                else
                {
                    strCheck = inXmlNode.OuterXml.Trim();
                    inTreeNode.Text = strCheck;
                }
            }
            tvXMLView.ExpandAll();
        }

As you can see, we are passing the node we are currently running and the tree view we are currently using. We check for it to contain children (indicating a parent node) and if it does, we create that node and then call the same function to go down through until we reach the child. Once the child is reached, we post the value of the child to the bottom node and then work back through. I also added a little bit of code to parse out the open and close brackets for the items in the event there is no value for that node. That about does it. That is all it takes. It can take this XML (found on Microsoft’s website as an XML sample) and parse it into a tree view.

<?xml version="1.0"?>
<catalog>
   <book id="bk101">
      <author>Gambardella, Matthew</author>
      <title>XML Developer's Guide</title>
      <genre>Computer</genre>
      <price>44.95</price>
      <publish_date>2000-10-01</publish_date>
      <description>An in-depth look at creating applications
      with XML.</description>
   </book>
   <book id="bk102">
      <author>Ralls, Kim</author>
      <title>Midnight Rain</title>
      <genre>Fantasy</genre>
      <price>5.95</price>
      <publish_date>2000-12-16</publish_date>
      <description>A former architect battles corporate zombies,
      an evil sorceress, and her own childhood to become queen
      of the world.</description>
   </book>
   <book id="bk103">
      <author>Corets, Eva</author>
      <title>Maeve Ascendant</title>
      <genre>Fantasy</genre>
      <price>5.95</price>
      <publish_date>2000-11-17</publish_date>
      <description>After the collapse of a nanotechnology
      society in England, the young survivors lay the
      foundation for a new society.</description>
   </book>
   <book id="bk104">
      <author>Corets, Eva</author>
      <title>Oberon's Legacy</title>
      <genre>Fantasy</genre>
      <price>5.95</price>
      <publish_date>2001-03-10</publish_date>
      <description>In post-apocalypse England, the mysterious
      agent known only as Oberon helps to create a new life
      for the inhabitants of London. Sequel to Maeve
      Ascendant.</description>
   </book>
   <book id="bk105">
      <author>Corets, Eva</author>
      <title>The Sundered Grail</title>
      <genre>Fantasy</genre>
      <price>5.95</price>
      <publish_date>2001-09-10</publish_date>
      <description>The two daughters of Maeve, half-sisters,
      battle one another for control of England. Sequel to
      Oberon's Legacy.</description>
   </book>
   <book id="bk106">
      <author>Randall, Cynthia</author>
      <title>Lover Birds</title>
      <genre>Romance</genre>
      <price>4.95</price>
      <publish_date>2000-09-02</publish_date>
      <description>When Carla meets Paul at an ornithology
      conference, tempers fly as feathers get ruffled.</description>
   </book>
   <book id="bk107">
      <author>Thurman, Paula</author>
      <title>Splish Splash</title>
      <genre>Romance</genre>
      <price>4.95</price>
      <publish_date>2000-11-02</publish_date>
      <description>A deep sea diver finds true love twenty
      thousand leagues beneath the sea.</description>
   </book>
   <book id="bk108">
      <author>Knorr, Stefan</author>
      <title>Creepy Crawlies</title>
      <genre>Horror</genre>
      <price>4.95</price>
      <publish_date>2000-12-06</publish_date>
      <description>An anthology of horror stories about roaches,
      centipedes, scorpions  and other insects.</description>
   </book>
   <book id="bk109">
      <author>Kress, Peter</author>
      <title>Paradox Lost</title>
      <genre>Science Fiction</genre>
      <price>6.95</price>
      <publish_date>2000-11-02</publish_date>
      <description>After an inadvertant trip through a Heisenberg
      Uncertainty Device, James Salway discovers the problems
      of being quantum.</description>
   </book>
   <book id="bk110">
      <author>O'Brien, Tim</author>
      <title>Microsoft .NET: The Programming Bible</title>
      <genre>Computer</genre>
      <price>36.95</price>
      <publish_date>2000-12-09</publish_date>
      <description>Microsoft's .NET initiative is explored in
      detail in this deep programmer's reference.</description>
   </book>
   <book id="bk111">
      <author>O'Brien, Tim</author>
      <title>MSXML3: A Comprehensive Guide</title>
      <genre>Computer</genre>
      <price>36.95</price>
      <publish_date>2000-12-01</publish_date>
      <description>The Microsoft MSXML3 parser is covered in
      detail, with attention to XML DOM interfaces, XSLT processing,
      SAX and more.</description>
   </book>
   <book id="bk112">
      <author>Galos, Mike</author>
      <title>Visual Studio 7: A Comprehensive Guide</title>
      <genre>Computer</genre>
      <price>49.95</price>
      <publish_date>2001-04-16</publish_date>
      <description>Microsoft Visual Studio 7 is explored in depth,
      looking at how Visual Basic, Visual C++, C#, and ASP+ are
      integrated into a comprehensive development
      environment.</description>
   </book>
</catalog>

Viola!

Note - I made the above form just for the purpose of this post

This simple tool has made a world of difference at work and everyone that supports the systems now uses it. I also made it so that you can right-click on the value and copy it to your clipboard so that you can have it without typing it as some of the values we are looking for can be long hexadecimal values. Then I added some preferences through a control panel that updates the registry and made this a full-featured tool. That concludes this post. Thanks for reading.

We started this project because of the ridiculous changes that have been made to a once highly respected tutorial site that caused it’s traffic and overall outlook to become terrible. They have let a lot of people down and there should be sites to counteract what was done. I am talking about Tutorialized. More about what they have done and where they went wrong can be read on John’s detailed blog post.

To make this site work, several things needed to be done. In this post, we are going to look at something that I wanted to be done automatically that you can pay to have done, but I felt this was something we could achieve for free. The feature I am talking about the ability to take a screenshot of a page and make it the thumbnail of a newly added site automatically with absolutely no intervention from us. On the main page of the site, there is a bar to add a URL of an RSS feed that we will use to scrape the content of your site and post them on our site automatically. Once a URL is entered, a row is put into a job queue table where the image creation begins.

The server this runs on is a command line only install of CentOS and does not have X windows or anything installed to produce graphical output. To achieve the browser view and take a screenshot, I used a virtual display using X Windows Virtual Frame Buffer This is a powerful tool that will allow you to create, control, and even view the graphical output via command line. You can use VNC over SSH to have visual remote control to a machine as well. Once this was configured on the server and I installed the necessary modules for firefox and some necessary font files we were ready to roll. I decided to use a cron job that calls the PHP interpreter and runs a PHP file to make like easier. The PHP file is in a secure location not access by the apache processes so it wont be accidentally called for any reason.

Some of the more interesting code in the PHP follows. We grab all the rows that currently need processed form the database and begin processing them. For each row, we run the following code:

 //get url from database
$url = $row['site_url'];

//Calls the virtual frame buffer and creates a virtual display and tells it what to run
$shellfile .= "DISPLAY=:3 firefox '$url' &\n";

//give the browser time to load
$shellfile .= "sleep 12\n";

//capture the screenshot of what is currently being displayed (id passed from database)
$shellfile .= "DISPLAY=:3 import -window root '/home/tutorial/public_html/avatar/$id.jpg'\n";

//wait for the image to be captured
$shellfile .= "sleep 3\n";

//kill firefox (don't want hundreds of instances running)
$shellfile .= "pkill firefox\n";

//wait for firefox to be killed
$shellfile .= "sleep 3\n";

//add final shell script line to kill the X virtual frame buffer
$shellfile .= "pkill Xvfb\n\n";

//compile the above strings into an actual file on the hard drive of the server
$shell = "autoimage.sh";
$fh = fopen($shell, 'w') or die("can't open file");
fwrite($fh, $shellfile);
fclose($fh);

//process the file via the shell interpreter
system("sh /jobs/autoimage/autoimage.sh");
sleep(3);

//delete the shell file
unlink($shell);

Note that the above code is all passed to a file on the hard drive and then processed by the shell interpreter and then deleted. This has then created the image in the images folder that we can use to show as the site thumbnail. We then run it through a resize and crop function and then update the file with the updated image. We verify the image is not black (something may have failed) and if it isn’t it is moved into the production database and ties to the new sites account.

This goes to show that with a little thinking outside the box and using the systems at your disposal, anything can be done. You just have to be willing to try new things. This system has worked flawlessly since implementation and has been a great success. Thanks for reading and look back for more updates.