The recommended filesystem for Squid on OpenSolaris is ZFS: http://wiki.squid-cache.org/BestOsForSquid
It is also recommended to disable the atime property on the filesystem holding the cache, and you may want to avoid using any type of RAID.
To achieve this on Nexenta (or OpenSolaris, whatever), first create the ZFS filesystem:
# zfs create -o atime=off -o mountpoint=/var/spool/squid3 syspool/squidcache
Then install Squid (here for version 3.x, as you might have noticed from the command). On Nexenta Core Platform 3 (NCP3):
# apt-get install squid3
To further improve the setup, use the aufs storage. To do this, just enable and update option cache_dir in /etc/squid3/squid.conf to read aufs instead of ufs (and further modify that line to best suit your real cache usage).