Posts in: Problems and outages
10/04/2008

We had a 2 hour outage due to a failure of the monitoring software. A minor error escalated to a large problem. It took an hour to get noticed and another hour to fix.

Unfortunately it was peak time for some of our customers and they missed out a few orders. Our fault. Guilty as charged. :-(

26/02/2012

Whoops. Had to stop the site to get some maintenance done. It should be only a few minutes, in case you noticed.

08/08/2011

AWS had a connectivity problem for a large chunk of their infrastructure – US-East-1 zone was unavailable for about 30 min. Most likely your web-to-print site was affected too. Bummer!

web-to-print connectivity

02/12/2008

Reports on your web-to-print orders already contain most of the information stored in the database, except the texts entered by users into web-to-print templates. It may be difficult to reconcile orders if they are placed by one person for the whole office. From now on, reports contain top 5 fields from every order added to the right of PayPal transaction columns. You may need to re-shuffle and merge some columns to add some sanity to the data if field names do not match across templates.

22/10/2012

Amazon is having problems with their storage system in N. Virginia data center, where most of our instances are located. It’s been going on for the last couple of hours and we hope they are fixing it.

In the meantime the site is down.

We are started relocation to a new zone, but it will take few more hours and result in some data loss. The recent transaction are likely to be missing.

14/11/2008

We are rolling out a new file management system. It started with deleting approximately 50,000 files it should never have deleted. Your files. Oooops ….

The good news, they are not lost. We just don’t know where they belong and it may take a few days to untangle. Any additional information will help. So, if you notice that any of your templates lost fonts, previews, thumbs or just stopped working – tell us straight away.

Most likely it’s a false alarm. Internal consistency checks show everything is OK, but better be sure than sorry. Full file matching takes ~ 72 hours so many of them there are. Pointing us at missing files will help understand the pattern and restore them quickly.

21/04/2011

Looks like we may have some downtime. AWS (Amazon Web Services) is having problems with storage.

Performance issues: Instance connectivity, latency and error rates.
Friday, 22 April 2011 02:40
In addition to the EBS volume latencies, EBS-backed instances in the US-EAST-1 region are failing at a high rate. This is due to a high error rate for creating new volumes in this region.

We see some of our instanced being next to useless with the number of errors they generate. It’s all in Amazon’s hands at the moment. Stay tuned.

26/04/2011

Our beloved hosting company, Amazon Web Services or AWS for short is still crook.

We are having constant disk failures across multiple servers. It sure impacts on the end users because the site produces a huge number of errors.

There is little we can do at the moment. Waiting for AWS postmortem and possibly changes to their SLA.

The promise of their service is high reliability of the storage and zone isolation. This recent outage shows that wasn’t the case. The outage spanned the entire region.

Jumping the ship and rushing to move elsewhere would be unwise. It’s unlikely Rackspace or some other provider is any better. What we need to do is add more redundancy and better contingency to the service.

 

15/03/2012

Amazon Web Services, is letting us down. From their RSS healthcheck feed:

2:40 AM PDT We are investigating connectivity issues for EC2 in the US-EAST-1 region.

All of our sites and servers were inaccessible for about 10 minutes.  It’s back to normal now.

14/04/2009

Here is a new layout, here it’s gone …

Several customers got very unhappy about the new layout. It didn’t work with their skinks and was too much a change for the type of users they have.

We’ll need few more days to enable the new layout with all the new features as an option that you turn on or off.

06/09/2009

We’ve been quiet on new features front in the last few weeks. The reason is the migration we will undertake some time this month from a physical data center where we kept our own servers to a virtual hosting environment in the US.

Today we are starting to clean up data so we don’t need to move all the terabytes of stale web-to-print files. The first lot to go are old orders and order files.

Every web-to-print catalog has a setting for data retention. The default for deleted web-to-print orders is 6 months. All other orders are retained for 13 months. They were never deleted. Not since some time in 2007.

Unfortunately your web-to-print site may need to be taken down for brief periods in the next 24 hrs. We picked the lowest activity time for this.

Let us know if the data we removed needs to be restored. We’ll put it into an archive for some time. It will be deleted permanently as we decommission our current servers.

10/09/2009

The web-to-print order deletion processes is more or less stable now. It was running most of the day today moving few hundred thousand files to a separate web-to-print storage. They will stay there for some time in case we moved what we shouldn’t have.

You may notice the order counts in some folders went down.

We couldn’t do without a few hiccups. They are being ironed out. Mainly it was about calculating the total printer balance and user shopping cart total. Stopped a handful of orders coming through properly. Sorry about that :(

Please, do not be shy to nag us if the site or some of the pages are slow. It may be some time before we notice. Same if any of the files you need are missing. E.g. it’s possible that some product previews got moved accidentally.

Email all problems to support@zetaprints.com or call if it’s urgent.

28/10/2011

Web to print functionality is pretty much unusable at the moment and will be for some more minutes due to an unexpected DB clean up. In other words, we were doing something we shouldn’t be doing at this time of day, screwed up and now trying to clean the mess while your customers are trying to order. Shame.

Sorry about it. :(

22/09/2008

We are running low on disk space due to a file replication problem. It was expected to be fixed by now, but looks more like another 6 to 8 hours before it gets any better.

Please, do not do any template or catalog maintenance work today, if possible.

Update: the performance should be back to normal now. Sorry about the inconvenience.

17/01/2012

Our servers have been going red-hot for a few days now due to increase in traffic. Everyone’s catching up on the lost time during the festive break. It would be all good if we could cope with it. Turns out our very scalable architecture has a little weak link in form of file replication and back ups. We made big changes to it over the New Year with a few hick-ups and stumbles, but it’s all settled now.

The problem arose when we started scaling the number of servers. It simply started choking up on cross-machine traffic and all the negotiations which file goes where. So adding more servers doesn’t add to better performance.

It is completely unacceptable and we are working on a fix, which may take a day or two to roll out. There will be brief outages few minutes long as servers are brought down for updates.

It’s a big FAIL for us, no excuses here.

Bah!

22/12/2008

We are moving servers around. It’s a bit painful on a live system, but once completed it will be worth a few minutes of downtime. We are adding more redundancy and more processing power, not just upgrading h/w.

We had an embarrassing 15 minute long downtime when a directory with critical files got wrong security settings and took that long to locate the problem and run an update.

Looks like we are going to have another 10 to 20 minutes of downtime now to do some more maintenance. Feel free to email your frustration to support@zetaprints.com. We’ll think of a compensation for the losses and discomfort.

Update: the downtime was 20 minutes. Approximately 20 users were cut off. The service should be back to normal. If you experience any problems with performance or error messages, please, email support@zetaprints.com for assistance.

11/11/2008

Our web-to-print software is due for a system upgrade. The whole cluster must be stopped for some time. Hopefully it will be seconds and no one notices. Most likely it will be minutes. We’ll keep you posted here.

ZetaPrints web-to-print system is fully hosted and runs on multiple servers (a cluster). The servers all talk to each to other about the state of cluster and to know who’s where and who’s got what. The piece of code that holds them together needs to be stopped. Once it happens they cannot communicate and the data will get out of sync pretty quickly. It’s safer to stop the whole cluster.

NZ, AU, HK and West Europe are not high users at the moment. Eastern Europe and Moscow web-to-print site are likely to be affected most. Sorry guys.

Update

The site is back to normal. Took us 30 minutes. Should have been faster. There was a tiny difference between staging and production environments and the change-over script failed.

Minor performance problems are quite possible until the servers figure out who’s who and get all in sync. Let us know if you experience any noticeable delays.

10/02/2012

One of our image processing servers lost the plot earlier today (February 02 2012). It was saying a font was installed when it wasn’t. As a result a lot of orders and previews got Arial or some other random font inserted in place of the proper font. It didn’t happen to all the orders, but enough of them to cause trouble.

The problem appears to be fixed now, but we can’t say for sure because it relies on a visual check, so please, let us know if any new orders are still faulty.

Regenerating the files should help. Send us a link to the order if it doesn’t.

Still trying to find out why this happened and if any other servers are affected.

02/01/2012

Happy 2012, a year of Dragon. We caught it’s flaming breath early on and got a bit scorched. In other words, the system went down on 1 Jan morning and stayed down for 21 hours while everyone was happily drinking / sleeping / diving /skying, or whatever our guys were doing while the DB server was trying to restart itself.

We know that the DB server stopped working after a successful backup. Just stopped. No errors, no messages in any logs – the process got terminated. The monitoring s/w was programmed to restart the server, but MS SQL wouldn’t start.

It was started manually many hours later without a hitch. That’s as much as we know for now.

Not a good look at all to start a year like this. Please, let us know if your site was negatively affected.

28/04/2011

We found several locations with replicated image files are are matching them to the originals. It is going to take time and we don’t know how many will eventually match.

AWS support haven’t given their final word whether we lost the data or not yet, so there is hope too.

It may be a good idea to re-upload images you need urgently, but it’s better to wait for the rest.

We’ll update you in a few hours on the progress.

24/10/2010

We had a problem with uploading files, which was quite disruptive and lasted for just over 4  hours.

Thank you everyone who alerted us. It is now fixed and the site is operating normally.

The problem appeared after AWS crashing some of our storage, but it’s us at fault because the recovery script didn’t run to plan and matters worse.

Sorry :(

03/05/2009

We had a major-major bug introduced on Friday.  It resulted in hundreds of web-to-print orders generated incorrectly.

Affected orders: Saved web-to-print orders where users made changes after saving.

Fault: the previews are correct, but the output files contain the data from the saved order before it was changed.

Action required: check all orders paced between Friday 1/05 and Monday 4/05 and email us links to any faulty ones.

The only way to fix the orders is to manually re-enter the texts. It is our fault and we will re-key faulty orders for you as soon as you give us a list. Please, mark any urgent web-to-print orders as such and we will process them first.

We fixed the bug with a temporary solution. It is still possible for users to place an order where the preview is different from the output if the order comes from Saved.  We are working on a permanent solution. Something should be out early on Monday. Please, keep checking all orders until then.

Faulty orders will not be billed and we will provide an equal number of free orders to compensate for the inconvenience.

13/04/2009

ZetaPrints web-to-print has no limit on order file size. We noticed that some templates produce unnecessarily large files. For example an invitation card comes as a 110MB download.

Please, keep an eye on the file size while downloading and fix any templates producing excessively large files. We will introduce a filter to temporarily block such orders, but would prefer you to take action first.

Email support@zetaprints.com to get help with reducing file sizes.

Once the filter is in place it will block downloading any files larger than a certain size (probably 20MB per A4 page). You will be able to fix the template, regenerate the output files and then download what comes out of it.

Some designs cannot be reduced in size because of all sorts of reasons. We’d like to see them and understand your requirements.  If you need it that large then no problem. It’s the post-card size designs 50MB each that worry us.

02/11/2009

How embarrassing. DoS followed by a really neat hack. We still don’t know what damage was caused.

Trying to bring the service online of the spare capacity.

02/12/2009

We are switching over from one host to the other right now. Your web-to-print portal is on the move.

The old web-to-print site displays a temporary page informing visitors why the site is down.

Once the new web-to-print site is up and the DNS has been updated you should continue using it as nothing happened.

Anyone accessing the old web-to-print site will be redirected to a page explaining how to update their DNS settings.

Check if any web-to-print files are missing after the switch-over.

Update: the site is up and running at the new location.

03/11/2009

We could not re-synch approximately 4 hrs of data prior to the crash.

You can access the files from orders from that period if the links are in the emails. Otherwise they will be restored later on today.

31/07/2013

One of the servers had a fault that went unnoticed for two weeks at the beginning of July. It was not saving files to the permanent storage while still reporting that everything was OK. By the time we realised there was a problem we lost 500 templates and 9000 images uploaded around that time.

We are trying to recover the files from other distributed locations, but many are lost forever.

web-to-print-404

Symptoms

Templates with no preview files, images with no thumbnails (just the name), orders with no previews.

What to do

Please, re-upload the missing files and regenerate orders.

Apology

Everything fails at some stage, but its is completely unacceptable that the file loss went for so long. We added more checks to the code for the future and the sysadmin responsible for the monitoring volunteered to spend a weekend working for a local Fair Trade shop.

Sorry, our bad :(

 

05/03/2009

Printing templates uploaded between 25 Feb and now need to be re-processed.

You do not need to re-upload them as they are already on the site.

This does not affect templates without printing templates.

Re-processing instructions

  1. Go to the template and click on fonts.
  2. Add any font from the list
  3. Remove the font
  4. Click Reprocess button

Sorry. It’s a real blunder. We are happy to re-process your templates if you have too many.

The change in the code was made to enable version control of uploaded template files. You will be able to roll back a few steps without re-uploading files. Another long awaited feature needed this change – ordering a saved order. An update with these features is on its way.

02/07/2009

We have a “small” problem with parsing new templates.

If a template has a layer with a variable image field in a power clip and no other fields at all the image will be ignored.

A fix is on the way and will be deployed in the next few hours. We’ll let you know.

Sorry :(

05/01/2012

Our email server IP address was blacklisted for about 2 hours for no apparent reason. We didn’t get any explanation, but it seems to be back to normal now.

Please, check your web-to-print orders on the site to make sure you didn’t miss anything.

 

19/09/2012

We have just fixed a problem with one of the servers that affected web-to-print file generation.

A lot of web-to-print orders were coming through without the files and had to be regenerated. Sometimes the regeneration failed too if it hit the same faulty server.

We fixed the problem. Please, click on Generate files in the order details they are missing and they should appear. Email us immediately if they don’t.

27/04/2011

We are back up and running after a 5hr long outage, but now on crutches.

And, all user and stock image are sitting on a drive AWS is trying to make available to us again. In other words, we don’t have them right now.

How it affects you

Most of user and printer images uploaded before the outage are missing. Their thumbs and small previews are still there, but the originals are on the missing drive.

If someone selects an image with a missing original it will come up blank on the preview and in the order file. There should be no errors generated.

This is confusing to users and is likely to result in a drop in orders.

What to do about it

Nothing a this stage. We are waiting for AWS to give us our data back. They only started working on it an hr ago. There must be a long queue of similar requests. :(

Why did it happen?

AWS started having issues with storage on Thursday. We noticed that some customer websites went down. Few hours later we started seeing the same problems in our servers. We held it together through the outage loosing instances and disks, but still having 100% of the data available and the sites running.

We tried to back up the data during the outage, but it simply didn’t work. AWS systems were too overloaded for this. We decided to play nice and didn’t persist.

Today AWS officially declared the outage over. The next moment the last alive disk with 1TB of user images started getting errors. 30 minutes later we lost some more critical data and had to stop everything to maintain data consistency.

How it was supposed to work

It just wasn’t supposed to fail all at once. Amazon has a motto “Everything fails”. They should add ” at once”. This is what happened.

Our data is replicated across multiple drives on independent servers running in different zones. If one goes down the data is still somewhere on another server. This strategy failed miserably in the current event of everything being down.

Conclusion

Jumping from AWS to maybe Rackspace is not going to make much difference. They may have the same extensive failure. Anyone can. We will be looking at replicating the data and keeping hot backup systems on different platforms. If AWS is down we run off Rackspace and vice versa. It is a major undertaking and a very costly one. Let the dust settle first.

We will update you on the situation with user and printer images in the coming hours.

10/02/2012

PDF, JPG and other print-ready files for orders older than a few days were showing as missing for quite a few orders.

The files were there all the time, but a bug was preventing their download.

Fixed now.

21/11/2011

We are having some weird connectivity issues. It is not specific to our web-to-print and dynamic imaging system. Many customer web-to-print sites are dropping connections for no reason as well. It manifests itself in half-loaded images, no CSS being loaded, broken pages and other unusual behavior.

Sorry, nothing we can do about it.Looks like it’s the network.

AWS health dashboard is saying it’s not their fault either.

Let us know if your site is having problems too.

12/12/2011

We’ve just rolled out an update to fix some bugs found in PDF generation when pages are left as default if not updated. The rumor has it that we botched it up big time. Web-to-print products with printing templates do not always produce the intended output. The PDF files do not always match the previews. It is bad and embarrassing, but we need another 8 – 12 hrs to fix it all properly. It’s out design fault because we didn’t account for one common scenario. It became apparent after the roll-out.

All faulty PDFs can be successfully regenerated when the problem is fixed in less than a day. We are working on it right now.

Shame. It was meant to be a merry post announcing a new cool feature :(

27/04/2011

We hid all pre-outage user images away from user libraries until we either get the data back from Amazon or know it’s not recoverable. There is still hope.

Missing images

Printer images

Printer and designer image libraries have all thumbnails and very few original images. If the missing images are used they come up blank on previews and in PDFs. We will either remove the missing one completely or restore them when we know the fate of the missing terabyte.

Stock images with templates

Some images are just assigned to templates and are not in any library at all. They continue to show on template / product pages, but will come up blank if the originals are missing. We will clean them up or restore together with printer and user images.

Faulty orders

There will be a fair amount of faulty orders with blank images and you may even be charged for them. Please, email us early next month with the total number of such orders. We will definitely wave the charges.

 

 

13/04/2008

Application of latest Microsoft patches to our Windows 2003 Server machines went terribly wrong. We are trying to bring them back to life.

Update: all back to normal.

The overall downtime was less than 2 minutes. Some users had to try more than once to get through.

29/05/2008

Yes, we did it again. The site was only partially functional for about 40 minutes today. There may be another scheduled outage later today, but won’t be longer than a couple of minutes.

Sorry :-(

14/08/2012

Someone pressed a wrong button a few minutes ago, ignored the warning and pressed OK to confirm.

As a result, all printer payments are showing as ‘2012-08-10’. We are working on restoring the data from a backup.

28/01/2013

All web-to-print orders paid via PayPal rely on a confirmation sent back to us from PayPal. It appears they stopped sending them about 20 hours ago.

Their system status update isn’t very specific, but lots of other sites posted similar notes.

Looks like their problems started on Jan 27 and are continuing.

Notification: Some impact to PayPal API – Jan 28

Update 4: Issue in receiving Instant Payment Notifications (IPNs) – Jan 27

Follow-up: Major Impact to Express Checkout, PayPal APIs, Virtual Terminal, and Website Payments Pro Payflow Edition (PayPal Payments Pro in US) – Jan 27

Web-to-print orders are being processed, but not released for download or moved into ToPrint folder without the payment confirmation from PayPal.

Web-to-print orders that do not require a PayPal payment are not affected.

24/11/2008

We are running much slower than usual today. Should be back to normal within the next hour or so. We have to switch search function off. It’s been misbehaving.

26/01/2009

We are being visited by aliens. This is the only explanation we can give to mysterious pink outlines on images and text. They appear randomly on newly uploaded templates. Existing templates don’t seem to be affected.

We need to trace this fault to its origins. So, if you get a preview with an extra outline, please, right click on the preview image and grab the URL of the preview JPEG file. Email the URL to support@zetaprints.com.

26/11/2009

Have you changed your web-to-print portal DNS record to a CNAME?

Change it now or contact us for help.

New data center

We are switching to a new data center some time this weekend. Tests are under way right now. Once we are happy and most of the data is synchronized we will change the IP address on zetaprints.com and your web-to-print site will be running off the new location.

A smooth ride is NOT expected. It’s quite possible we will need to switch back if the new hosting is not working to the plan.

Every switch means a bit of lost data. Although we will automatically synchronize both locations it takes minutes or even hours for all the new files to transfer.

Possible web-to-print data loss

Do not upload new templates until an announcement is made that we have moved, if you can wait. If the template can’t wait it’s not a big deal to upload. Just be aware it may disappear for some time if it ended up on a wrong server until it is found and placed where it should be. Please, let us know if you are missing any orders, images or templates on Monday.

02/12/2008

Update: We had a word from the ISP the outage was canceled.

ZetaPrints system will be switched off for up to 2 hours on Sat/Sun for maintenance.

Our ISP will be changing the power wiring in the rack and there is no way we can have the servers running at the same time.

Expected time

NZ/AU: early Sunday

Europe: Saturday afternoon

N/America: Saturday morning

23/01/2009

Our hosting provider will be re-wiring power supplies in a few hours. The site will be completely off-line during that time. Sorry, it’s beyound our control.

Update: the site has been switched off for ~ 20 minutes.

Update: the site is back up. Report any problems to support@zetaprints.com or call for urgent assistance.

16/09/2009

Power’s down. Trying to figure out what’s happening.

18/05/2009

We had a problem with mismatches between web-to-print previews and the output files. It was happening when a user clicked back in the browser and ordered from an outdated page. We made changes and a warning if a user tries to do that.

Users may get a message saying “Please, check the latest previews again” when their page is outdated.

Another known problem is with saved orders. If a user saves a web-to-print order and then edits any of the images used in it then the final order will contain the new images and the preview has old ones. We’ll be adding a patch to fix this problem later today.

We also added a host of new features. More on them later.

Please, keep an eye on your order files today and let us know if we missed anything.  Call emergency support line if you don’t get a quick reply via email.

01/12/2008

We have a problem with cloning catalogs and some other features requiring file copying. The bug is being fixed right now, but it will take a couple of hours before everthing is back to normal.

Update: templates and catalogs can be copied now, but re-preocessing templates after assigning fonts still has problems.

27/04/2011

This outage was bad enough. Bad on Amazon, but also bad on us. In the end it’s our fault – assume nothing, trust no one. We trusted in AWS promise to keep zones separate and the core services running. Lesson learned. Here are a few ideas we want to implement in the near future.

Missing terabyte

Option 1

There is still hope we’ll get our missing terabyte of data back and quickly restore all missing images. Unfortunately Amazon is less than responsive at the moment, but they promised to look into this. It would be the best solution.

Option 2

We may need to ask you to re-upload your images, but this would be done via FTP as a bulk upload. You will simply upload everything you think may be relevant. Our software would crunch the images and match them against the database. No need for you to manually reassign them. Still, an unnecessary headache for you.

We will remove missing images from the database after some time or on your request to stop them appearing altogether.

There will also be a special web page to see what images are missing.

Data storage reliability

Backups

We don’t believe in them. A back up is only as good as the last restore. There is always data loss with backups anyway.

Replication

All printer data will be mirrored to a different type of storage (EBS/S3) within Amazon at no additional cost to customers. It will effectively double our costs, though. A hit we are willing to take.
We will offer an option of replicating all data to a different provider like Rackspace at additional cost.

Hot mirrors

We will maintain a ready to go mirror of the software with a different provider, probably Rackspace in case AWS fails again.

Moving so much data between 2 different systems on a short notice when everyone is probably trying to the same is not going to be feasible. Only those paying for off-site replication will be able continue with minimal interruption. All the others will be able to continue, but the downtime will be longer with some data loss.

No-w2p ordering

Our magento web to print extension will allow a fall-back option of taking orders without a preview in case the system is down.
All the fields will be populated and images uploaded, but the artwork would need to be prepared manually.

This is already in the development plan.

Faulty orders

We will wave fees for any faulty orders. Email us early in May with the list. We will not run the billing cycle until well into May.

Thank you for your tolerance and support

We had quite a few phone calls and emails. Everyone is frustrated, but no one freaked out or yelled at us. We’d better go start working on the solution again.

27/04/2011

We still don’t have our lost terabyte back.

Users see their images in the image library, select them and get blank space instead.

We will temporarily hide user images from their libraries and then start a gradual restore process from cached data on instances that survived the outage. Users will have to upload images again if they want to use them while we are trying to restore the data.

02/08/2013

Multi-page web-to-print templates can have text and image fields shared between multiple pages.

The user can change it only on one page where it is encountered for the first time. All the other pages in the same web-to-print template will reuse the previously entered value.

We had a problem with our web-to-print engine updating pages it didn’t need to update when the shared fields were involved. If an update failed the user had to start over. This problem has now been fixed.

There is still a performance limitation. If a field spans too many pages with a lot of images it may take too long to update them all resulting in a time-out. Please, let us know if you encounter this situation.

 

30/11/2009

We are still testing the new location and synchronizing new files. The switchover is planned for approximately 6am GMT on Tue.

The site is likely to go down for about an hour or two. This is how long it takes to re-sync the database. Having less downtime will increase risk of data going missing.

Please, do not upload new templates, fonts or stock images until we are running off the new location, unless they can’t wait.

11/05/2011

We discovered that approximately 10% of image we managed to restore are corrupt.

These image are moved to “Missing-“[original name] folder for printers. User images with corrupt files are hidden from view.

We have most of them in different other backups and already started restoring them. It will take days because we have to match them by their MD5 hash.

12/03/2009

We’ll be rolling out some security packages in the nezt hour or so. The system will be taken offline for a few minutes.

26/04/2011

Thanx Amazon.

We’ll keep you posted.

We are up, but on crutches. Took us 5 hrs to restore most of it.

06/09/2010

We are rolling out an update to all our web-to-print portals. There is a chance some web-to-print templates may generate incorrect PDFs. In particular, there may be a mismatch in spot colors, CMYK <-> RGB conversion, missing fonts, missing image, etc.

Please, review all orders before printing even if the templates performed flawlessly before.

Do let us know immediately if you spot any issues.

04/05/2009

Please, stop printing and check all multipage files against previews. Only multipage designs are affected.

Orders coming trough Re-Order or Complete Order will most likely have one or more pages generated incorrectly.

We suspect there is a problem with one of team members committing wrong code into production. We are trying to roll back to the last stable build, but it may take a couple of hours.

Update: we are back to normal. The problem was eliminated by rolling back to a stable build.

04/12/2008

Some of our servers were updated with latest Microsoft patches last night, about midnight. Everything went smoothly, no one noticed, until this morning when some templates failed to process. To make things worse, it affected the way optional elements are removed. An unknown number of optional elements were left on pages when they had to be removed.

We do not know how many orders are affected. They are still being affected, but we are working on a patch right now. It is expected to be fixed within next 4 hours.

Please, check all orders before sending them into production.

We will post instructions how to fix them after uploading the patch. Otherwise you can email support@zetaprints.com with a list of order IDs or URLs and we’ll fix them up for you.

Unfortunately, this type of error was not logged and only visual checks would pick it up. Hopefully users did pick it up if it happened to them.

Sorry :-(

Update: the problem appears to be fixed, but we are still monitoring. Let us know if your templates are misbehaving.

28/08/2011

We are rolling out a large upgrade to some of the core software and making changes to the hosting infrastructure at the moment. It makes your web-to-print site a bit unresponsive. The performance will fluctuate during the upgrade, which is expected to last for a few hours. We will update you on the progress.

Fingers crossed.

 

27/07/2010

We had an outage on www.zetaprints.com. You wouldn’t even notice if it wasn’t for the CSS file for web-to-print skins we keep there.

A few of customers’ web-to-print portals depend on the CSS files and refused to function. We removed the web-to-print skins from the pages, so the browser wouldn’t even try to load them for the time of the outage. As a result, your web-to-print site looked a bit boring with our default skeleton design for about an hour.

www.zetaprints.com is back online and we put the skins back onto your sites.

I’m tempted to point the finger at the hosting company. Not in this case. We could have a better back up strategy. It actually worked, but not as quick as one would hope.

Sorry :(

12/06/2010

A lot of emails sent to Yahoo mail users get delayed by Yahoo and then blocked.

There is absolutely no reason for them to do so as we only send transactional emails about web-to-print orders and registration info. Our IP address is not listed in any spam databases either. No good as it’s hurting web-to-print sales of our customers. :(

We contacted Yahoo about this, so hopefully it will get resolved in the coming hours.

In the meantime, it may be a good idea to follow up customers with @yahoo.com emails personally.

We’d like to see all non-delivery reports, if you can be bothered.