platform/manifests
1
1
Fork 0
Code Issues 7 Pull Requests 6 Actions Packages Projects Releases Wiki Activity

Loki-write is failing because it can't delete file from NFS #8

New Issue
Closed
opened 2025-09-29 06:25:56 +00:00 by mrtz-j · 11 comments
mrtz-j commented 2025-09-29 06:25:56 +00:00 (Migrated from gitlab.com)
Copy Link
Copy Source

Upon startup, the loki-write pod always creates an empty WAL file, which it attempts to remove during shutdown. When the new pod starts and accesses the same PVC/storage, it crashes immediately because it can't delete the WAL dir.

It seems to be a NFS file:

image

Related issue from upstream ref

Upon startup, the `loki-write` pod always creates an empty WAL file, which it attempts to remove during shutdown. When the new pod starts and accesses the same PVC/storage, it crashes immediately because it can't delete the WAL dir. It seems to be a NFS file: ![image](/uploads/bdd8aa165bcbd2969729ea49c8172262/image.png) Related issue from upstream [ref](https://github.com/grafana/loki/issues/13923)
mrtz-j commented 2025-09-29 06:25:57 +00:00 (Migrated from gitlab.com)
Copy Link
Copy Source

set status to To do

set status to **To do**
mrtz-j commented 2025-09-29 06:26:41 +00:00 (Migrated from gitlab.com)
Copy Link
Copy Source

changed the description

changed the description
mrtz-j commented 2025-09-29 06:29:11 +00:00 (Migrated from gitlab.com)
Copy Link
Copy Source

changed title from Loki is failing because it can't delete NFS file to Lok-writei is failing because it can't delete file from NFS

<p>changed title from <code class="idiff">Loki is failing because it can&#39;t delete <span class="idiff left right deletion">NFS </span>file</code> to <code class="idiff">Lok<span class="idiff left addition">-write</span>i is failing because it can&#39;t delete file<span class="idiff right addition"> from NFS</span></code></p>
mrtz-j commented 2025-09-29 06:29:34 +00:00 (Migrated from gitlab.com)
Copy Link
Copy Source

changed title from Lok-writei is failing because it can't delete file from NFS to Loki-write is failing because it can't delete file from NFS

<p>changed title from <code class="idiff">Lok-write<span class="idiff left right deletion">i</span> is failing because it can&#39;t delete file from NFS</code> to <code class="idiff">Lok<span class="idiff left right addition">i</span>-write is failing because it can&#39;t delete file from NFS</code></p>
mrtz-j commented 2025-09-29 06:31:58 +00:00 (Migrated from gitlab.com)
Copy Link
Copy Source

assigned to @mrtz-j

assigned to @mrtz-j
mrtz-j commented 2025-09-29 06:32:01 +00:00 (Migrated from gitlab.com)
Copy Link
Copy Source

set status to In progress

set status to **In progress**
mrtz-j commented 2025-09-29 08:38:48 +00:00 (Migrated from gitlab.com)
Copy Link
Copy Source

The pod is back up after manually deleting the file and removing the old PVC. Will keep the issue up on lower priority just in case.

The pod is back up after manually deleting the file and removing the old PVC. Will keep the issue up on lower priority just in case.
mrtz-j commented 2025-10-04 08:02:34 +00:00 (Migrated from gitlab.com)
Copy Link
Copy Source

unassigned @mrtz-j

unassigned @mrtz-j
mrtz-j commented 2025-10-04 08:02:37 +00:00 (Migrated from gitlab.com)
Copy Link
Copy Source

set status to On hold

set status to **On hold**
mrtz-j commented 2025-10-25 20:51:38 +00:00 (Migrated from gitlab.com)
Copy Link
Copy Source

set status to Done

set status to **Done**
mrtz-j (Migrated from gitlab.com) closed this issue 2025-10-25 20:51:38 +00:00
mrtz-j commented 2025-10-25 21:07:01 +00:00 (Migrated from gitlab.com)
Copy Link
Copy Source

mentioned in merge request !44

mentioned in merge request !44
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Compat/Breaking
Breaking change that won't be backward compatible
 Kind/Bug
Something is not working
 Kind/Documentation
Documentation changes
 Kind/Enhancement
Improve existing functionality
 Kind/Feature
New functionality
 Kind/Security
This is security issue
 Kind/Testing
Issue or pull request related to testing
Priority
Critical
1
The priority is critical
Priority
High
2
The priority is high
Priority
Low
4
The priority is low
Priority
Medium
3
The priority is medium
Reviewed
Confirmed
1
Issue has been confirmed
Reviewed
Duplicate
2
This issue or pull request already exists
Reviewed
Invalid
3
Invalid issue
Reviewed
Won't Fix
3
This issue won't be fixed
Status
Abandoned
3
Somebody has started to work on this but abandoned work
Status
Blocked
1
Something is blocking this issue or pull request
Status
Need More Info
2
Feedback is required to reproduce issue or to continue work
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
bast (radovan) juselius (Jonas Juselius) mrtz (Moritz Jörg) oletytlandsvik (Ole Tytlandsvik) release-bot renovate-bot simkir (Simen Kirkvik) stigrj (Stig Rune Jensen)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: platform/manifests#8
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?