We have a large FreeNAS server that is loaded with files. I am looking for advice on the best way to get things cleaned up, and I know there’s tons of duplicates.

File Types:

• Images
• Text
• Videos

File Counts:

• 10,000,000+ Files
• 200+ TB

I’ve tried running many other duplicate scanners, but they haven’t been easy because the scanners crash when they get logs too big, it’s hard to get context, and it takes days to scan without checksums (Takes a really long time to checksum (MD5) files). And to top it off, they only run on one PC so I can’t even enlist the rest of the team to help clean up.

I need a way to make it so that we can easily scan files, identify duplicates, and be able to ideally save scan results and checksums such that we don’t need to keep re-scanning the same files again and again. I like beyond compare, but it helps after the duplicates have been identified.

What do you guys do to scan this much data and make sense of it / organize it?

We have an onsite NAS that everything backs up to nightly. Not large, about 3-4 terabytes. I’m looking to get an identical NAS for a satellite office that I can push deduplicated backup to each night as well. I’m not sure the best solution for this. We use NovaStor to do our nightly file level backups, and they suggested I talk to Buffalo since that’s who they recommend.

I’d prefer to check with the community first. I understand cloud solutions would be cheaper/easier/etc but that’s not what we’re looking for.

What you want is an encryption/crypto mitigation tool. Our MSP provides one as part of their service. Most of these viruses first add a file to the infected location with the decryption advice (eg DECRYPT_FILES.html). The mitigation tool looks for these files on file shares and disables the share if detected. It gives us a chance to find out the user that created the file, take them off the network, then spin up the share again. Then we can take our time to clean up the endpoint.Your backup plan should include cold and hot storage solutions. We use shadow file copies, a local backup appliance which copies to the cloud several times a day and month-end snapshots of important file servers.I've lived through 4 crypto-scenarios. A couple of other key takeaways. YMMV. Block external email clients if you are able. Our business does not need you to have access to Yahoo mail. Since we are not providing spam filtering on your personal email, we shut it down. KnowBe4 training. AV/Antimalware rarely catches it. Your users are your biggest threat. Train them. KnowBe4 is very affordable and invaluable. Block USB devices. Look into Safend/Wave DataProtect. If you don't need USB devices to do your job, you can't plug in your compromised thumb stick. User Permissions. Cryptolocker viruses can only infect files the user has access to. Audit your file share permissions to make sure people don't have owner access carte blanche. Use a tool like ManageEngine or Netwrix to check for users with too much access. Network port access. Do what you can to keep personal devices off your network. Use MAC filtering or just unpatch unused ports when not in use. Make sure your company policy defines what is allowed on the company network. Audit regularly. Wifi - Create a guest SSID with no network access for personal devices if you require it. Segment traffic with a VLAN or firewall rules.

The Synology RS820+ and RS820RP+ are the same hardware but the latter has two power supplies (RP = Redundant Power).

Probably not a huge requirement for an offline backup

For my understanding of deduplication it in a way leaves a bookmark of every place that file applies. It should be seamless to the end user. Keep in mind, this would on happen across common files, not all files.

To the end user they wouldn’t see any changes on their end.

From Microsoft

• Group Shares: Group shares include regular file shares that a team might use for storing data and includes environments that use Folder Redirection to seamlessly redirect the path of a folder (like a Documents folder) to a central location.

learn.microsoft.com

Archived MSDN and TechNet Blogs

I have put in several Netgear Ready NAS units - use veam endpoint back to back up the pc’s and server to the NAS - then from ready NAS you can backup to various cloud services. This provides onsite and offsite backups. Works great.

Worked with one business years ago - had a poor setup server with no RAID and poor backup - ended up having to send server disk off to a recovery service - $7,000.00 dollars later they got 95% of the data back - hard lesson to learn and the NAS and cloud storage cost are nothing if you value your data.

Lets backup a bit and get the 10,000 foot view.

There are two issues:

1. How to actually perform a backup and restore

2. How to get data to and from wherever it is being safely stored

For 1, most people use a commercial service/product that keeps track of what data to backup, scheduling, writing files to backup media, logging, restoration, etc. Some of this is determined by your needs- do I just want a copy of my data, or do I need immediate disaster recovery if my server room catches fire? You can do any or all of this yourself but for something business-critical, a vendor with a tested solution is going to be easier and safer for most of us.

For 2, you have a choice. Some sort of networking (LAN or Internet, depending on where your files are backed up to) or physical delivery of backup media. A small business might backup to tapes or hard drives and have someone take them home every night. A large 24/7 business might have an entire duplicate data center at a remote location, ready to switch over if needed.

Obviously, cost varies considerably

My advice is to figure out what you need, both in terms of irreplaceable data and in terms of restoration time. Figure out what your budget is and then start talking to vendors to see if those things all line up. Also, talk to your insurance company and see what kind of disaster/business continuation coverage you have.