The Synology RS820+ and RS820RP+ are the same hardware but the latter has two power supplies (RP = Redundant Power).

Probably not a huge requirement for an offline backup

We have a large FreeNAS server that is loaded with files. I am looking for advice on the best way to get things cleaned up, and I know there’s tons of duplicates.

File Types:

• Images
• Text
• Videos

File Counts:

• 10,000,000+ Files
• 200+ TB

I’ve tried running many other duplicate scanners, but they haven’t been easy because the scanners crash when they get logs too big, it’s hard to get context, and it takes days to scan without checksums (Takes a really long time to checksum (MD5) files). And to top it off, they only run on one PC so I can’t even enlist the rest of the team to help clean up.

I need a way to make it so that we can easily scan files, identify duplicates, and be able to ideally save scan results and checksums such that we don’t need to keep re-scanning the same files again and again. I like beyond compare, but it helps after the duplicates have been identified.

What do you guys do to scan this much data and make sense of it / organize it?

Hi All,

We are in the process of evaluating vendors for a new SAN. We currently have a mix of Compellent and Equallogic. The vendor has given us pricing based on a hybrid array with an amount of storage that we feel might be a little light. Raw storage will be about the same as what we currently have. They are saying that 2:1 dedupe is very likely and actually on the conservative side. I’m curious about what dedupe ratios other Nimble owners are getting so I can tell whether this is an accurate statement? The SAN will be used primarily for VMFS volumes.

Also, what’s the prevailing thought on running Windows deduplication on a VM that resides in a SAN volume with dedupe on it? Our Windows server VMs already have the dedupe running and we are seeing pretty good results with it. Just wondering if it will be an issue with the SAN based dedupe.

Thanks for your input.

For my understanding of deduplication it in a way leaves a bookmark of every place that file applies. It should be seamless to the end user. Keep in mind, this would on happen across common files, not all files.

To the end user they wouldn’t see any changes on their end.

From Microsoft

• Group Shares: Group shares include regular file shares that a team might use for storing data and includes environments that use Folder Redirection to seamlessly redirect the path of a folder (like a Documents folder) to a central location.

Archived MSDN and TechNet Blogs

As always money usually rules the day, if you use an in house nas then replicate it to an another site (nas to nas), QNAP has good facility to do that without using any third party software. I am sure there are other NAS devices that do the same, but is a product i am familiar with. Or you can use another rapid recovery core that sits offsite connected to a another NAS but the price can become dear quick but at least its all upfront costs; will your internet connection support the movement of the amount of bandwidth that you will consume during your backups? deduplication and compression will reduce it but if the amount of data changes significantly daily that can add up quickly.

I would assume this is virtualized environment? (i know assumptions!) If it is would you consider a high availability setup, where you would have a standby virtual server sitting at another location and rapid recovery exports the images nightly. (still a local data repository)

There are also third party data repositories you can replicate your images to (azure,aws rackspace) again money money money. (storage and bandwidth consumption)

I’m sure you will get many more suggestions as there are some studly backup types in this community.

good Luck

we are in process to evaluate the replacement to our file share Windows Storage SAN

to new Storage …

so any one could give recommendations with NAS netapp and EMC

to get use of deduplication and compression and how much the ratio i could get for unstructure data

and how this will be compared to windows duplication ??

FWIW I found the Dell PowerVault and NetApp E series SAN’s to be practically the same. The hardware setup (iSCSI) was similar and the management software for provisioning storage (RAID, virtual disks, host mappings) was similar, in fact exactly the same. As @oscaroneeye mentioned, understanding the fundamentals is the most important. The vendor specific stuff you will learn on the job, and even those will likely be based on fundamental technology.