Accurate identification of grain cultivars is critical for improving crop yields, streamlining agricultural workflows, and ensuring global food security. Near-infrared (NIR) spectroscopy offers a rapid, non-destructive solution for grain classification. However, its effectiveness hinges on extracting meaningful spectral features. We propose SpecFuseNet, an attention-enhanced residual autoencoder, as a lightweight deep learning model for extracting NIR spectral features and classifying grain varieties. The encoder integrates Fused Efficient Channel Attention (FusedECA) and a Spectral Residual Gate (SRG) to extract informative spectral features, while a mirrored decoder enables robust spectral reconstruction. This architecture supports both spectral reconstruction and cultivar classification, with robust performance and minimal complexity. We evaluated SpecFuseNet on three NIR datasets: barley (1,200 samples, 24 varieties), chickpea (950 samples, 19 varieties), and sorghum (500 samples, 10 varieties) using stratified 5-fold cross-validation. The model achieved classification accuracies of 89.72%, 96.14%, and 90.67%, respectively, outperforming PCA-based machine learning models (SVM, Random Forest, XGBoost) and deep learning baselines such as standard Autoencoder (AE) and Convolutional Sparse Autoencoder (CSAE). These results demonstrate SpecFuseNet's potential as a fast, interpretable, and deployable solution for real-time classification in field-based and resource-limited settings, with a lightweight design that enables deployment on portable or smartphone-connected NIR spectrometers, supporting sustainable and precise agricultural practices.