The problem with CSV import plugins

The quickest route to linking a web store to a supplier is with an e-commerce plugin. They are installed into your store and usually need CSV data files from your supplier. Some have the ability to grab data from FTP, run on a regular schedule, and do other basic things.

Unfortunately they won’t grow with your business.

Plugins are meant to cover the lowest-common-denominator. They can do the basics but then will hit a wall as soon as you need to create a custom pricing rule, or change the product meta-data, or simply change how the inventory is updated. Plugins are created for the purpose of selling in volume and therefore the developers who create them are not interested in your custom requirements. However, any substantial e-commerce store will need customization.

Some of the more common questions I see are:

– What about my ERP system? I want to relay WooCommerce orders back to my supplier and they need them uploaded via FTP or with a web service.

– My supplier doesn’t use CSV. It is XML format. How can I load this into my Shopify store?

– How can I get my supplier’s shipping data back into my PrestaShop store so that my customers can use the tracking number.

– My supplier’s data feed doesn’t provide variants but I need them to show different t-shirt sizes. How can I fix this?

– Can you import products into my store categories for me? They don’t match my supplier but I can tell you which ones to map.

These are all exceptions which we handle on a daily basis at Data Here-to-There. We can adapt our innovative platform to any supplier or online service.

It’s surprising how many data feeds out there have low-quality product information. When I first got started integrating supply feeds for my clients I was shocked at how bad the data was. These are, after all, companies which *want* to sell their products. Yet they have terse (or missing) product descriptions, missing images, and so on. Data Here-to-There can actually enhance data feeds – create variants, add tagging, automatically find product images using Google, or even web-scrape entire product catalogs if your supplier doesn’t have a data feed!

Plugins are a great tool to get started with and can be sufficient for a simple single store, single supplier setup. But to grow your e-commerce empire you’ll need customized automation that fits your business strategy.

How to make a great data feed

If you are a drop-shipper, you need a data feed.

A Data Feed provides a catalog of products that merchants can import into their store. Maybe it’s a collection of kids toys with the SKU, title, description, price, and other information that the store owner can use to entice potential buyers. Data feeds are a critical component of any e-commerce supply chain. So how do you make a data feed?

To get started you need a list of products. Perhaps this is simply an Excel spreadsheet, or maybe it is stored in your QuickBooks system or in an inventory management database. The first step is to decide which products can actually be drop-shipped. There might be products like gift cards or rare products that you don’t want to supply. So those need to be filered. Now we create key information for the vendors. We need the following information.

SKU – stock keeping unit. This is an unique identifier which will be used when placing orders.
Title – The name of the product.
Description – A friendly explanation of the product which commonly includes HTML formatting.
Cost – The cost to your vendor, they will mark it up based on this number.
MSRP – Manufacturer’s Suggested Retail Price. This is higher than cost and used to indicate sale prices.
MAP – Minimum Advertised Price. Sometimes you want to set a lower limit on the price. Vendors can’t sell below it.
Stock – How many units are available to be sold? Vendors use this to prevent oversells in their store.
Manufacturer – The brand name of the product.
UPC – Sometimes referred to as GTIN, UPC codes are required for most major marketplaces.
Weight/Height/Length/Width – Determines shipping costs
Images – Pictures showing the product from different angles. These are stored separately.

The above data is required by most web stores and marketplaces so it’s good to provide as much as possible. It’s also important not to have empty values in any of these fields because it can break their store. Missing prices, titles, or other data causes a lot of grief because the vendors won’t be able to list them.

The most common data feed is simply a set of CSV (Comma Separated Values) files that vendors can download. CSV files are very easy to create, in fact you can make them using Excel. If you have a large inventory it makes sense to split your data feed into multiple CSV files. Especially the stock numbers – these fluctuate the most! So if you have a stock CSV file that will allow your vendors to grab it quickly and update their stores frequently without having to download *everything*. It saves your bandwidth costs too.

The Cadillac of data feeds doesn’t stop here. We can create a CSV file for product variants as well. This allows us to group collections of similar products. For example, a watch which can have a blue, green, or red background. It’s the same watch just with different colors. So we call these ‘variants’. The vendor has to be told how to connect up these variants though so this data is critical. Otherwise they have to list each watch as a separate product and that looks bad.

It’s also a great idea to create attributes as well. Descriptive tags help vendors sell products because tags can be searched in their store. For example a book seller can create an author tag, number of pages tag, an ISBN tag and so on. This data can be imported and allows customers to search by author, number of pages, etc. What a great way to help vendors out!

It’s also handy to publish a list of any products that you removed. This provides a signal to the vendor that they have to remove those products from their store. Otherwise they will end up selling products that no longer exist.

Data Here-to-There can design your data feed for you. We can use your source data and transform it into standard web-store compliant CSV data files. These files are then downloaded by your vendors and imported into their store. We’ll automate it as well so that it stays fresh every day.

Create your own marketplace

Marketplaces are really hot right now. On the freelancer sites I see daily requests to help create new ones. Marketplaces for fitness equipment, used goods exchange, health foods, and more. It makes a lot of sense because a marketplace owner can create a robust products collection by aggregating offerings from other vendors. However, I see one common flaw in those systems.

Most often the marketplace owner requires the vendors to import data on their own. It usually means submitting CSV files into a magical portal or manually updating products one-by-one. Not a lot of fun, right? Sometimes these vendors are not very technical so asking them to do this is a big job. It also puts the accuracy of the marketplace inventory in their hands. What happens if they get busy and forget? The marketplace will oversell which will frustrate customers.

A better way is to use automation. If the vendor already has a web store, just connect it to the marketplace. New products can be added automatically, existing inventory and pricing can be refreshed, and products which are no longer being sold can be removed. The vendor simply manages their own store and instantly has the benefit of exposing his products into a new market. Their store has essentially become a drop-shipper to the new marketplace.

If a vendor does not have a store already, why not use their existing work process? Let them use their own Excel spreadsheet format or connect to their QuickBooks system. This allows them to focus on running their store while the marketplace increases its product set.

Data Here-to-There specializes in Marketplace automation. We can connect vendors to your WooCommerce, Shopify, Magento, or any other e-commerce platform. Products are imported while you sleep and kept fresh. We can also relay orders from your marketplace to your vendors, integrate shipping, or a variety of other online services.

Stores that we like

As a company that automates web stores, we see a variety of different platforms. It’s like walking into a Starbucks and asking for “coffee.” It’s never just “coffee.” It’s frappy, frothy, mocha, mega, grande, and that’s before we get to the sweetener options. Except in our case we see Magento, WooCommerce, Shopify, PrestaShop, and lots of lesser known animals (I’m intentionally leaving out the marketplaces because that’s a whole story of its own). Each one has its own set of versions from beginning to end. Some have simple security and others are like Fort Knox. Some are easy to use and some act like a misbehaving child…

Us: Please add this picture to the Golden Boots, product 101.

Store: OK…no, wait! Error 505

Use: What is error 505?

Store: Hahah!

Use: (sigh) Please add this picture to product 101

System: OK!

This is a (slightly exaggerated) real life story based on a Magento integration. Using the Magento REST API we would weave product data into a beautiful catalog. Except once in a while it would just throw random errors. The solution was to drop the REST API entirely, eat a whole bottle of Aspirin, then code a solution that writes directly to the database. We have a lot of emotional scars from Magento.

If I was to set up a store for myself it would use WooCommerce. It’s like the 8′ kid that always gets picked first for basketball. WooCommerce has an awesome API, smart user interface, amazing upgrade facilities built-in, and a smart but not too smart security model. I used to think Shopify was the Cat’s pyjamas until I got under the hood with WooCommerce.

It comes with an API written by people who know how to write an API. If I need to list a set of products they make it easy. If I need to authenticate, that’s easy too.

Shopify is like that too because it’s based on the same engine. One thing Shopify has over Woo is the built-in attribute for a Manufacturer name. It’s a simple thing yet has such a big impact. Pretty much every supply feed has manufacturer names associated to products. When I integrate a Shopify store I can simply map it right over…voila! With other systems I have to reinvent the manufacturer name ‘wheel’.

PrestaShop’s XML output is pretty cool. I like it because I can grab a big chunk of products fast. Most systems use paging with ridiculously low limits which creates a pile of network traffic.

Are you choosing a platform right now? If you want to be able to automate it in the future make sure it has an API. Most systems use a REST (Representational State Transfer) API. At some point in time you’ll want to link up QuickBooks, a supplier (or multiple suppliers), shipping services, your Aunt Betsy’s coupon service, etc. Today I found out that it’s actually possible for an e-commerce platform to *not* have an API. SquareSpace promotes itself as an e-commerce platform but provides no way to integrate inventory, shipping, or anything else. Yikes!

Product Curation

Every store has a personality. Brightly colored kids toys, sleek and sexy swimwear, or maybe some tasty looking chocolates. When we go shopping we create an association between a niche of products and the store that sells them. The power of habit brings us back and quickly that place becomes our ‘go-to’ store.

A long time ago I went shopping for a pair of glasses. I was used to paying about $300 every two years because that’s what they had always cost. But then I heard from a friend that you could get the same style online for $25. That’s amazing! They sell glasses. Just glasses. Not contact lenses, not laser surgery, and they make it easy to see what you’re buying. They have focused their product line.

Product curation is the process of selecting products which fit a merchant’s niche. It is very strategic and defines the store. In the good old days a merchant would go to the supplier and buy products, box them up, and bring them to his store to sell. But today this is much more virtual. We have drop-shippers and marketplaces with lots of button clicking.

There are many innovative ways to select products from supply data feeds.

– Choose products by categories. Perhaps the merchant has a supplier who provides all kinds of clothing – kids, women’s, men’s, and chic pet cardigans. A simple form of curation is to choose the categories which will look good in his store.

– Choose specific products. Sometimes there are very specific products so we can select them by their title or SKU.

– Choose by keywords. Each product has a description which contain valuable information. Keywords can be used to narrow in on specific products. Perhaps the merchant wants all ‘garden’ related products.

In a future article we will look at what it means to automate Product Curation. How can we choose products on behalf of the merchant?

What is an “Oversell?”

One of the common problems in running an e-commerce store is the oversell. It happens when a customer buys something which is not in stock. All web stores and marketplaces track inventory but it is only as good as the data that the merchant provides. If a web store believes that there are 1000 Purple Plastic Cupcakes in stock but the merchant sold the last one an hour ago, any further sales will be oversells. The store will continue to oversell and promise customers prompt delivery even though this is clearly not possible.

The consequence is a lost sale. The merchant tells the client that the item can not be delivered as was originally promised. The customer goes elsewhere to buy his fake pastries. Perhaps they get frustrated with the experience and continue to buy goods from a competitor.

Sometimes this isn’t the only penalty. Some drop-shippers will penalize the merchant’s account when an oversell occurs. Most marketplaces also track this kind of problem and will score it against the merchant’s record. It works against the merchant’s long term reputation.

A simple solution to this problem is to ensure proper synchronization between the merchant’s store and the supplier. For a simple store with a few products this isn’t really necessary. But for larger stores where there are hundreds of products and frequent changes to stock, it is critical.

Our system processes thousands of products per day from a wide variety of suppliers. This ensures that our clients have fresh inventory numbers and avoid the dreaded oversell.

Abandoned Products

If an online product falls in the forest…

One of the most overlooked problems in e-commerce is what we call “Abandoned” products. These are products which at one time existed in a supplier’s data feed. Imagine a kitchen furniture supplier who has a Butcher Block Family Table product. It sold very well for a number of months but then it was removed. However, any merchant who uses the data feed has already placed this product on their store. How can a merchant tell if he has products in his store which can no longer be fulfilled?

It’s like an oversell waiting to happen. The supplier can’t provide this product any more yet it is on the merchant’s site. Eventually someone is going to click there.

Some drop-shippers provide a separate data feed of retired products. This can be used to weed out any abandoned products in the store. However, it is quite rare to find this in a supply feed. Even with that information it is still only relevant to merchants who actually listed those products.

Abandoned products are part of a broader service that we call Catalog Synchronization. The product catalog is comprised of all the possible supply data merged with the store’s product set. New products can be added dynamically, changed product data is updated, identical products are left alone, and abandoned products are hidden or removed based on the merchant’s policy.

Catalog Synchronization goes hand-in-hand with inventory synchronization. Both processes are critical to running an efficient e-commerce store.

Dynamic Pricing

Pricing is a critical part of an E-Commerce store’s strategy.

Well tuned prices will bring in customers. Bad pricing will turn them away. Customers are always looking for great deals, suppliers are looking to preserve their minimum advertised prices, and stores need to make a profit. That’s why vendors need to have a pricing strategy. They need to make a profit and compete in the online marketplace or else there is no point selling anything. We need to find the right mix between competitive pricing and long-term sustainability of the store.

Suppliers provide two prices – the warehouse cost and the suggested retail price (MSRP). Some will also provide a minimum advertised price (MAP) and other price related options. The store owner can simply choose to list all of their products using the MSRP. Since the MSRP is higher than the warehouse cost, the store owner will have a profit margin built-in. Using the MSRP will pretty much guarantee that the store isn’t competitive. This is due to the fact that every other drop-shipping vendor is doing the same thing. So what can be done?

Data Here-to-There provides a wide range of pricing strategies. Here is a quick overview.


This is the simplest approach. We use the MSRP from the drop-shipper for each product. It comes with a built-in profit margin but is not very competitive.

MSRP Adjust

We can discount MSRP values based on a dollar amount or percentage. This allows the vendor to undercut other stores which are simply using the de-facto MSRP value. Prices do not appear ‘on sale’, they simply show a lower value. It applies to all products uniformly.

Standard Margin Pricing

This option is the most popular among our clients. They choose a margin (percentage) that they wish to add to the warehouse cost of each product. Here is an example where the store owner has chosen a margin of 1.25 (25% above cost).

Item ID / SKU = BK13934
Name = Peter’s Favorite Recipe Book
Cost = 44.47
MSRP = 59.95

Using the Cost field of $44.47 and a margin of 25% we end up with a store price of $55.59. The MSRP is $59.95 so the store will show ‘on sale’, discounted by $4.36. The same rule will apply to all products. The margin will always be preserved because it is added to the warehouse cost. If the supplier increases the warehouse cost, the system will automatically increase the price accordingly.

Category Based Pricing

Some categories can be set up as ‘loss leaders’ to entice customers into the store. Other categories can be higher. This is a good strategy for selling accessories. For example, we can price printers at a lower margin than printer ink.

For this approach we assign a default store margin to catch any products that aren’t categorized. We then set margin for each category which differs from the default margin. For example, printers and plotters should be set to 10% which printer ink and toner has a margin of 25%.

Category based pricing gives the vendor more control over pricing.

Competitive Pricing

This is one of our more advanced pricing strategies. We can design a system which pulls prices from competitor web stores. For example, we might find the following prices for “Peter’s Favorite Recipe Book.”

Walmart shows a price of $62.33
Indigo Books shows $68.34
Mary’s Books shows $59.95

We can set it up to take the lowest price (or the average price) then subtract a fixed amount. For example the lowest price is $59.95. If we subtract $1 the store price would be $58.95 which is lower than all of the other stores.

We can also set up a lower limit so that the system never goes below a minimum margin. This ensures that the competitive pricing does not drop below the vendor’s comfort level. So taking our previous example with a margin of 25%, if a competitor drops below $55.95 this store will not chase it down any further.

We can integrate third party pricing feeds or web scrapers for this purpose.

Price Decoration

Data Here-to-There can ‘decorate’ prices to a common value. You’ve probably seen stores where prices always end in .99 or .95. We can do this for any store.  For example, $14.23 would become $14.99 if you set a ‘.99’ price decoration.

Tiered Pricing

Sometimes it makes sense to adjust the price based on the wholesale rate.  Perhaps the prices range so much that a general store margin provides too little profit on the low-priced items.  We can create ranges for each margin based on the wholesale price.  i.e.

  • If the wholesale price is between $0 – $25, add 65% margin.
  • If the wholesale price is between $25 – $50, add 45% margin.
  • If the wholesale price is between $50 – $100, add 25% margin.
  • For everything higher, add 15% margin.

We can create multiple ranges and margins to suit the supply data.

Custom Pricing

In addition to the above strategies we can also develop completely custom solutions – integrate with an online pricing service, create custom rules, synchronize prices across multiple stores or markets, and much more.
Regardless of the strategy chosen, Data Here-to-There ensures that prices never drop below a special safety margin. No matter what happens, products will not be sold at a loss. We also ensure that any minimum advertised price (MAP) restrictions from the supplier are enforced. In addition, we also handle bad data problems which should never happen (but do!) where MSRP is lower than cost, MSRP is not available, cost is not available, etc.

Pricing is a very strategic part of an E-Commerce store therefore our system has been built to be flexible. Contact us today to discuss your E-Commerce automation needs.

D&H Dropship Images

D&H provides great products.  They publish a data feed that we use to keep our client stores fresh.  It contains the product titles, descriptions, prices, and other things that are typically needed for a catalog.

But it has a common problem – missing images.

The FTP location ( has a standard repository that dropshippers access to get the latest images.  There is a single image per product that you can match by SKU.  Taking a random look at the thumbnails we can see there are lots of placeholders:

Lots of “Image Not Available” and Lenovo brand logo images.

How can you sell products with placeholder images?  Or completely missing images?  It’s not easy.  Most people want to see what they are buying.

If you are using Shopify the Image Finder app can help you to find matching product images and import them into your store.  We can also provide the same thing for WooCommerce and other carts.

Optimized WooCommerce Hosting

WordPress and WooCommerce can be very heavy.  We’ve seen stores with 30,000+ products which grind to a halt when using shared hosting.  Pages load slowly, the API connections drop out, or the store is just very slow to respond when a customer is shopping.  Some hosts do not understand WordPress very well and don’t dedicate the right resources to supporting stores.

We’ve partnered with the Fullhost team to deliver optimized WooCommerce stores:

Click here to see the plans

We run our automation processes in the same data centers which means that your store will benefit from the tight integration.