JPanel fills entire JScrollPane

I'm making a level editor for a project of mine so I added a JPanel to a JScrollPane in order to view the level being worked on. The idea was that if the level was too big the scroll pane would handle that. The issue is when the level is to small, when I try to size the panel to a size smaller than the scrollpane it gets stretched to fill the whole scrollpane instead of the size I set. I can't seem to find a solution to the problem.

Setting up the JScrollPane:

LevelView levelView = new LevelView();
JScrollPane scrollPane = new JScrollPane(levelView);

JPanel:

public class LevelView extends JPanel {

    private static final long serialVersionUID = 1L;

    public LevelView() {
        Dimension dim = new Dimension(Editor.getLevel().getWidth() * Editor.UNIT, Editor.getLevel().getHeight() * Editor.UNIT);
        setPreferredSize(dim);
        setMaximumSize(dim);
        setMinimumSize(dim);
    }

    @Override
    protected void paintComponent(Graphics g) {
        super.paintComponent(g);
        System.out.println(getWidth() + " " + getHeight());
        g.setColor(Color.BLACK);
        g.fillRect(0, 0, getWidth(), getHeight());


        // Draw grid
        g.setColor(Color.WHITE);
        for (int x = 0; x < getWidth(); x += Editor.UNIT) {
            g.drawLine(x, 0, x, getHeight());
        }
        for (int y = 0; y < getHeight(); y += Editor.UNIT) {
            g.drawLine(0, y, getWidth(), y);
        }

    }

}

Thanks in advance!